inspire jamaica - unicef
TRANSCRIPT
INSPIRE Jamaica: Screening and Early Intervention System for Children and Families
INSPIRE Jamaica:
Early Intervention Screening
Project and Referral
Final Report
Grant no: UWI UNICEF 914
9/19/12-1/31/14
Early Intervention Program
University of Oregon
Jane Squires, Principal Investigator
1
INSPIRE Jamaica: Screening and Early Intervention System for Children and Families
Introduction to the Final Report
History Since its establishment in 2003, the Early Childhood Commission (ECC) in
Jamaica has worked strategically with partners and stakeholders to identify priorities and an
action plan for development of a comprehensive, effective early childhood system focused on
achieving optimal development for all Jamaican children. Early Child Development (ECD)
sectors, including Health, Education, and Social Security, identified an Effective Screening and
Early Intervention System for Children at Risk as the critical foundation of this system.
Development of valid and reliable screening tools was a critical first step for building this
early childhood system, with three focal components: 1) Screening and documentation system to
identify households at risk, 2) Screening and documentation system to identify young children at
risk for developmental or behavioral delays and 3) School readiness screening to determine
whether children have the necessary skills to learn and succeed in primary school and to identify
preschool children at risk. These tasks were the focus of INSPIRE Jamaica, a collaborative
project in partnership with the Government of Jamaica including the Early Childhood
Commission and agencies serving young children and families, UNICEF, children and families
in Jamaica, the Early Intervention Program at the University of Oregon, and the University of
West Indies.
This final report summarizes the research conducted as part of the INSPIRE Jamaica
project, including the history of the development and adaptation of measures for each system
component, as well as the data collection, analyses, and results of psychometrics studies of these
measures. Finalized protocols for screening tools and user’s guides were developed and are
included to facilitate accurate administration of the tools in health and educational settings in
Jamaica.
Acknowledgements University of Oregon researchers would like to acknowledge the
contributions of the University of West Indies research team for their work and dedication to
Project INSPIRE. Many thanks to Maureen Samms-Vaughan for envisioning the development
of this system of supports for children and families in Jamaica and her on-going contributions to
this project. Special thanks to Sydonnie Shakespeare for leading the on-site data collection in
Jamaica. Sydonnie and her team of research assistants, Simone Lee, Nara Anderson-Figueroa
2
INSPIRE Jamaica: Screening and Early Intervention System for Children and Families
and Allana Ingram, remained positive despite numerous data collection challenges. This
research never could have happened without their hard work and dedication to project INSPIRE.
Finally, we would like to thank the service providers, families and children in Jamaica that
participated in Project INSPIRE. We hope the tools developed will further the system of
supports for young children and families in Jamaica.
3
INSPIRE Jamaica: Screening and Early Intervention System for Children and Families
Table of Contents
Chapter 1: Technical Report Introduction………………..………………….5
Chapter 2: Family Risk Screening Technical Report…………..……………7
Chapter 3: Child Development Screening Technical Report…..…………..18
ASQ-Jamaica Technical Report……………………..……….18
SWYC Technical Report……………………….…………….36
Chapter 4: School Readiness Screening Technical Report………….……..51
Appendix A: Utility Survey………………………………………………..69
Appendix B: Child and Family Information……………………………….71
Appendix C: References……………………………………………………73
4
INSPIRE Jamaica: Screening and Early Intervention System for Children and Families
Chapter 1: Introduction
In 2012, researchers from the University of Oregon’s Early Intervention Program and
from the University of West Indies collaborated to develop culturally appropriate, valid and
reliable screening tools for use in Jamaica’s early childhood screening system. The screening
system consists of three main components: family support and risk, child development, and
school readiness. See Figure 1 for an outline of the screening components of the INSPIRE
Jamaica Early Childhood Screening System, and proposed administration schedule.
The Family Support Screening tool (FSS), originally developed by the Early Childhood
Commission in Jamaica after significant research on family risk factors, has been reformatted
and revised. The purpose of the FSS is to gather information about family and household risk in
order to provide families with needed support and resources. The FSS asks basic demographic
questions about families, along with questions about parent-child interactions, family health and
lifestyle, and family safety at home and within the community.
The child development component aims to identify those children “at-risk” and/or those
identified as having concerns as part of developmental and behavioral surveillance used during well-
child visits in Jamaican clinics. Two tools, both developed in the United States, were identified and
researched for use with children and families in Jamaica. The Ages and Stages Questionnaires
(ASQ) is a parent-completed child-development screening tool that screens a child’s development in
communication, motor, cognitive and personal-social areas of development. As part of Project
INSPIRE, an adapted version of this tool (ASQ-Jamaica) was developed with input from Jamaican
parents and service providers about the cultural appropriateness of the tool. In addition, the Survey
of Well Being of Young Children (SWYC), which screens for developmental and behavioural risk,
was researched for use with Jamaican families.
Finally, the purpose of the School Readiness tool is to screen children for child development
and readiness for primary school at the end of the four-year preschool. The school readiness
screening tool contains three sections: the ASQ-J child development screening tool for 4-5 year old
children; the Child Behavior Rating Scale; and the Jamaica School Readiness Screening Skills.
The psychometric properties of these screening tools were investigated as part of a
research grant funded through UNICEF. Research questions, utility and psychometric studies for
each of the screening tools are described in detail in the following report.
5
INSPIRE Jamaica
Screening and Early Intervention System for Children and Families
Figure 1. Components and proposed Administration Schedule for the INSPIRE Jamaica
Early Childhood Screening System.
6
INSPIRE Jamaica: Screening and Early Intervention System for Children and Families
Chapter 2: Family Risk Screening
Introduction and Tool Development
The Family Support Screening (FSS) tool was designed to gather information from families in order to
assess family and household risk and to provide families with needed support and resources. The FSS
tool is intended to be completed by professionals or paraprofessionals (e.g. health providers, teachers,
PATH Social Workers) through an interview with a main caregiver, an adult who resides with the child
and is responsible for decisions regarding the child’s care and wellbeing. The FSS tool includes five
sections, with between six and eleven questions in each. Each section of the FSS tool is listed below;
sections are described in detail in the Family Support Screening User’s Guide.
Section 1: Demographic and Socio-Economic Profile of the Household
Section 2: Welfare of Children in the Household
Section 3: Household Health and Lifestyle
Section 4: Family Safety in the Household and in the Community
Section 5: Parental Stress and Parent-Child Interaction
Development of the Family Risk Screening. The Family Support Screening Tool (FSST) was first
developed by the Early Childhood Commission in Jamaica after significant research on family risk
factors. In the Fall of 2012, researchers from the University of Oregon reviewed the most recent
version of the Family Support Screening tool and made some formatting changes, including
eliminating scoring guidelines, to create a pilot version of the tool. Before beginning the pilot phase,
these researchers conducted a focus group in Kingston, Jamaica on the pilot version of the FSS tool
with parents, teachers, Jamaican stakeholders, and University of West Indies (UWI) research
assistants. Feedback provided during this focus group was generally positive. The FSS tool was
revised in the following ways based on focus group feedback.
Section 5: Parental Stress and Parent-Child Interactions had been cut from an earlier version of
the tool and was added back into the pilot version.
A separate Antenatal version of the tool was created for use in antenatal clinics; this antenatal
version includes initial questions relevant to pregnant women.
Individual items were revised for clarity and appropriateness; this included adding in examples
and making subtle changes to wording of items.
In order to guide referrals during the pilot phase, a question was added to each section of the
tool that inquires whether families would like support or assistance to address concerns in that
particular area.
7
Finally, an interviewer guide was created to provide further guidance and clarity on particular
questions for practitioners.
Research Objectives and Questions
One of the key objectives of Project INSPIRE was to study the psychometric properties of the
Family Support Screening. Research was divided into 2 phases; Phase 1 focused on the utility of the
ASQ-J for Jamaican children and families. Phase 2 focused on descriptive statistics, reliability and
validity studies. Following are research questions for each phase:
1) Phase 1. Research Questions
a. What is the utility (including cultural sensitivity) of the Family Support Screening for
families and service providers in Jamaica?
b. What modifications need to be made to the Family Support Screening?
2) Phase 2. Research Questions
a. What are descriptive statistics (e.g., means, standard deviations) of the Family Support
Screening for Jamaican households?
b. What is the validity of the Family Support Screening (e.g., sensitivity, specificity) when
compared with other psycho-social measures, such as the Center for Epidemiological
Studies—Depression Scale (CES-D), (Radloff, 1977); Parenting Stress Index (PSI),
(Abidin, 1983); and Ages and Stages Questionnaires: Social Emotional (ASQ: SE).
Phase 1. Utility Data Collection on the Pilot Family Support Screening
During the Pilot Phase, UWI research assistants completed the FSS tool through interviews with
parents attending well child check-ups at health clinics and parents of children in preschool classrooms.
Parents provided utility feedback on the clarity, appropriateness and meaningfulness of FSST items.
One hundred utility surveys were collected on the FSS tool. Table 2.1 summarizes FSS tool utility
data. A copy of the utility survey can be found in Appendix A at the end of this technical report.
8
Table 2.1
Summary of Responses to Utility Statements*
Total Strongly
Agree
Agree No
Opinion
Disagree Strongly
Disagree
Missing
N N (%)
N (%)
N (%)
N (%)
N (%)
N (%)
In general,
questions clear
and easy to
understand
100 47 (47%)
49 (49%) 2 (2%) 0 (0%) 0 (0%) 2 (2%)
Questions
appropriate for
child and
family’s culture
100 33 (33%) 63 (63%) 1 (1%) 2 (2%) 0 (0%) 0 (0%)
Completing tool
provided
meaningful
Information
100 17 (17%) 59 (59%) 13 (13%) 9 (9%) 1 (1%) 1 (1%)
Would like to
use this tool
again
100 11 (11%) 69 (69%) 8 (8%) 10 (10%) 1 (1%) 1 (1%)
*Response to statements (e.g., In general, were the questions clear and easy to understand?) were on a
5 point Likert Scale, (Strongly Agree, Agree, No Opinion, Disagree and Strongly Disagree).
Additional Written Comments on Utility Surveys:
Add open questions.
First ask questions-if there is crime and violence in the community, if person says no, give
examples
This tool would have taken less time to complete if I had assistance.
Question needs to be disguised so that it doesn’t make people uncomfortable.
The tool could look at questions such as whether the child is engaged versus withdrawn. If
parent can cope.
Extend the tool to much more schools.
The tool should improve on more questions about parent.
University of Oregon researchers conducted a second round of focus groups with parents and
UWI research assistants after the FSS tool had been piloted. Participant feedback in these focus groups
9
was overwhelmingly positive, though a few suggested specific item changes. In addition, the following
advantages and disadvantages were shared by parents in the focus group.
Advantages of the tool:
o Good to get conversation started and getting information out there.
Disadvantages:
o People might hesitate to answer questions but desire for help could motivate people to
answer.
o Questions about wanting support or assistance could be more generally different for the
whole household as opposed to for the main caregiver.
Summary of Utility Findings Utility data on the FSS tool were quite positive. Almost all--96%
of respondents—strongly agreed/agreed that the questions on the FSST are clear and easy to
understand. Ninety-six percent of respondents strongly agreed/agreed that questions were appropriate
for their child and family’s culture. Seventy-six percent of respondents strongly agreed/agreed that
completing the tool provided meaningful information about the child; Eighty percent of respondents
strongly agreed/agreed that they would like to use this screening tool again. Current findings and
anecdotal reports from the UWI research team suggest that the FSST will be an appropriate and useful
screening tool for determining Jamaican family risk and needs.
Phase 2. Psychometric Data Collection on the Family Support Screening
The second phase of data collection involved collecting psychometric data on the field version
of the FSS questionnaire. Research staff from the University of West Indies assisted in identifying
health clinics and preschool classroom settings that provided as well-stratified a sample of Jamaican
families and children as possible, given financial and logistical constraints of the project. All
parents/caregivers in targeted settings were provided with information about the child and family risk
screening projects and asked if they would be willing to participate. As part of participation in the
research, main caregivers received an FSS questionnaire from the teacher or researcher along with a
form asking for the child’s demographic information and a research consent form. The FSS
questionnaire was completed through an interview with a research assistant. The research assistant
assisted the parent with referrals to community support services as needed. Procedures ensuring
protection of human participants were approved by the University of West Indies and the University of
Oregon institutional review boards and were followed in all research phases.
The following data are based on 251 FSS questionnaires completed by parents of children
10
between 1 and 66 months of age.
Population Sample for the Family Support Screening
The data analyses that are contained in this report are based on 206 completed questionnaires.
Each parent or caregiver who completed a questionnaire was asked to complete a demographic form.
Demographic data included information on the child’s age, gender, parish the child resides, setting
screening took place, who completed screening, and number of family household resources (as a proxy
of risks). It should be noted that there were 103 missing demographics, so percentages reported are
based on the 148 data for which demographic information was collected. In some cases, individual
item data was also missing. The demographic data for the population sample are displayed in Tables
2.2 through 2.6. A copy of the demographic form can be found in Appendix A at the end of this
technical report.
Gender of child and parish where child resides. As shown in Table 2.2, the gender distribution
for the sample was 59.9% male and 40.1 % female. Table 2.3 contains data on the parish where the
child lived at the time of the data collection. The majority of children lived in Kingston (41.9%), St.
Andrew (35.8%), or St. Catherine (12.4%) parishes. A small percentage of children (1.4%) resided in
other parishes in Jamaica.
Settings Where Screenings Took Place. Table 2.4 contains data on the setting where the FSS
screening took place. A large percentage (61.5%) of FSS screenings occurred in health settings, 32.4%
occurred in educational settings, and 6.1% took place in “other” settings. Approximately 76% of these
settings were public and 22% were private settings. Ninety-six percent (98.7%) of screenings were
conducted in urban settings (4.1% of the total sample was considered “inner” urban), while only 1.4 %
of screening occurring in rural settings. See the Limitations of Study section for a discussion of the
imbalance or urban/rural settings and recommendations for future research on this tool.
Table 2.2
Gender of Children
Frequency Valid Percent
Male 85 59.9
Female 57 40.1
Total 142 100.0
Missing 109
11
Total N 251
Table 2.4
Setting Where Screening Took Place
Person Completing Questionnaires. Table 2.5 contains data on the person completing the
questionnaires. The majority of individuals completing the questionnaires were mothers (92%). It
should be noted that the majority of these (98.6%) were completed with the support of a research
assistant. While FSS can be completed independently, data on the FSS tool was collected primarily by
research assistants through an interview format. Further research would need to be conducted to
determine how much support is required in actual health and educational settings in Jamaica to
complete the FSS tool, and whether main caregivers are able to complete it independently.
Table 2.3
Parish Where Child Resides
Parish Frequency Valid Percent
Kingston 62 41.9
St Andrew 53 35.8
St Catherine 31 12.4
Other 2 1.4
Total 148 100.0
Missing 103
Total N 251
Setting Frequency Valid Percent
Antenatal 15 10.1
Clinic 76 51.4
Early Childhood
Educational 26 17.6
Primary School 3 2.0
Nursery 19 12.8
Other 9 6.1
Total 148 100.0
Missing 103
Total N 251
12
Table 2.5
Person Completing Questionnaire
Household resources (child risk). Data on the number of household resources was collected as
a proxy of child risk status. Parents were asked to self-report household resources from a list of 26
resources (e.g., telephone, car, water heater). A total number of resources was entered for each family;
data was not entered in a way that identified specific resources held by each family. Household
resource data are displayed in Table 2.6. A little more than two percent (2.1%) of the sample reported
having between 1 and 4 household resources, 12.8% reported having between 5 and 8 resources,
24.8% reported having between 9 and 12 resources; 33.3% reported having between 13 and16
resources; 22% reported having 17-20 resources, and 5% reported having more than 21 household
resources.
Cutoff Scores and Percent Identified of Population for the Family Support Screening
A main goal of Phase 2 was to establish cutoff scores for the Family Support Screening (FSS)
tool in order to provide guidance to practitioners in identifying families with a high number of
concerns and in initiating referrals and other support. A discussion of validity studies follows.
However, validity samples were not adequate to inform the establishment of cutoff scores. Instead,
cutoff scores were derived using the normative sample (N=251; Ante-natal N=20); cutoff scores were
determined based on percentage of families identified.
Frequency Valid
Percent
Mother 136 92
Father 6 4.1
Grandmother 2 1.4
Aunt 2 1.4
Other-family 2 1.4
Total 148 100.0
Missing 103
Total N 251
13
Table 2.6
Total Number of Household Resources
# of Resources Frequency Valid Percent
1-4 3 2.1
5-8 18 12.8
9-12 35 24.8
13-16 47 33.3
17-20 31 22
21-24 7 5
Total 141 100
Missing 110
Total N 251
Initially researches established an overall cutoff score of 8 total concerns on the FSS tool. This cutoff
identified approximately 16 % of the normative sample. Jamaica may decide to use this overall cutoff
score. However, researches recognized that the intent of the FSS tool was to identify and follow up on
family risk factors. Because the overall cutoff score did not identify individual FSS sections with
higher concern scores, researchers created separate cutoff scores for individual sections that identified
approximately the same percentage of the normative population per section. As illustrated in Table 2.7,
each individual section of the FSS tool was assigned a unique cutoff score that would identify between
8 and 10 percent of the normative population. These cutoff scores are included in the summary sheet
on the final page of the FSS tool, and instructions are provided in the User’s Guide for using these
section cutoff scores to guide follow-up with families. Overall, the percentage of families in the
normative sample who were below the cutoff score in one or more of the individual FSS sections was
27%. Thus, if the normative sample is representative of the Jamaican population, providers can expect
to identify approximately 27% of families using these cutoff scores.
As discussed above, the section cutoff scores were initially established in order to identify a
particular percentage of the population. Researchers also analyzed the data to determine mean scores
and standard deviations within each section. Using means and standard deviations, cutoff scores are
set at two standard deviations above the mean score for each section, as illustrated in Table 2.8. Cutoff
scores are similar when computed based on percent identified and means/standard deviations.
14
Table 2.7
FSS Tool Cutoff Scores
Section N Cutoff Score Percent of Normative
Sample Identified
A (Ante-natal) 20 3 15
2 251 4 8.4
3 251 2 9.2
4 251 4 9.2
5 251 3 8.4
Table 2.8
Mean Total Concern Scores, Standard Deviations, Cutoff Scores, and Percent Identified by Section of
the Family Support Screening
Section 2 Section 3 Section 4 Section 5
Valid 251 251 251 251
Missing 0 0 0 0
Mean Total Concerns 1.38 .47 1.41 1.32
Standard Deviation 1.33 .77 1.32 .90
Cutoff Score 4 2 4 3
Percent Identified 8.4 9.2 9.2 8.4
As indicated above, Phase 2 research also included a plan to collect an FSS validity sample
using three corroborating tools: the CES-D (Center for Epidemiological Studies-Depression Scales),
the PSI (Parenting Stress Index) and the ASQ: SE (Ages and Stages Questionnaires Social-Emotional).
A validity sample size of 50 was targeted for all three tools. The total data collected included the 33
PSI, 28 CES-D, and no ASQ:SE.
Researchers analyzed correlations between total scores on the PSI and total concerns on the
FSS tool and found an overall correlation of .235. This correlation was weak and not significant.
Researchers also analyzed correlations between total scores on the CES-D and total concerns on the
FSS tool. Because eight of the CES-D scales had items for which responses were missing, researchers
calculated mean scores to replace missing item responses. The overall correlation between the FSS and
the CES-D was .459. Though this correlation is significant at the .05 level (2-tailed), it is also
15
considered weak. Correlations are illustrated in Tables 2.9 and 2.10 below.
Table 2.9
Correlations between FSS Total Concerns and PSI Total Stress Score
FSS Total Concerns PSI Total Score
FSS Total Concerns Pearson Correlation 1 .229
Sig. (2-tailed) .200
N 36 33
PSI Total Score Pearson Correlation .229 1
Sig. (2-tailed) .200
N 33 47
Researchers also analyzed the agreement between FSS cutoff scores and PSI and CES-D total scores.
For this purpose, two respondents who exhibited defensive responding on the PSI were eliminated
from the sample, resulting in a sample size of 31 for this analysis. Researchers looked at agreement
between those identified by the PSI and those families who had one or more area above the cutoff on
the FSS. The agreement was .74. However, no one in the validity sample was identified by the PSI as
having stress, this agreement does not provide meaningful information.
The fact that no one in the PSI sample was identified as having stress could be due to the fact that this
sample was not randomized, to the fact that participation by families was optional and/or to the fact
that the surveys were administered by unfamiliar researchers rather than by health providers known to
the families. Future research efforts might include a randomized sample representing a broader range
of families.
Limitations and Future Directions. There are several limitations to this study that should be considered
when implementing the Family Support Screening and considering future research directions. It is
important to note that this was an ambitious study, conducted within an exceptionally short time frame
and with limited resources. As discussed earlier, demographic information was not collected from 103
families, so percentages reported are based on the 148 families from whom demographic information
was collected.
16
Table 2.10
Correlations between FSS Total Concerns and CES-D Total Score
FSS Total Concerns CES-D Total Score
FSS Total Concerns Pearson Correlation 1 .459*
Sig. (2-tailed) .014
N 28 28
CES-D Total Score Pearson Correlation .459* 1
Sig. (2-tailed) .014
N 28 49
* Correlation is significant at the .05 level (2-tailed)
Data was also missing for some individual questions on gathered demographic forms. Because
demographic data is missing, it is not clear whether the normative sample is representative of the
general Jamaican population. Collected demographics indicate that the majority of data was collected
in three parishes: Kingston, Saint Catherine and Saint Andrew. As also discussed earlier, 98.7% of the
FSS screenings for which demographic information was collected took place in urban settings, while
only 1.4 % took place in rural settings. Future studies on the FSS tool should include a representative
sample of the Jamaican population.
As also discussed above, the small size of the validity sample combined with the weak and
insignificant correlations between chosen concurrent validity tools required researchers to use a
different method to establish cutoff scores. Cutoff scores were derived based on percentage of families
identified and presumed a normative sample that was stratified in terms of number of risk factors, and
included households with a low, medium and high number of risk factors. Because the FSS tool covers
a broad range of risk factors and areas of potential family distress (health, violence, stress, community,
parent child interactions etc.) it was challenging to identify concurrent validity tools that measured
similar areas. It may be helpful for future studies to conduct a factor analysis to identify categories of
risk factors covered by the FSS tool and to examine concurrent validity using additional measures that
have been used or validated in Jamaica and that correspond to the identified categories.
17
INSPIRE Jamaica: Screening and Early Intervention System for Children and Families
CHAPTER 3: Child Development Screening
Introduction and Tool Development
Project INSPIRE research staff studied the utility and psychometric properties of two child
development screening tools--one with a developmental focus, the Ages &Stages
Questionnaires: Jamaican version (ASQ-J), and one that is recommended to be used when social-
emotional/behavioral concerns are identified, the Survey of Wellbeing for Young Children
(SWYC). This chapter provides information on utility studies and psychometric studies for both
tools. The first section covers technical information on the ASQ-J, followed by the SWYC
technical report.
Part I
ASQ-J Technical Report
This section offers a range of technical information about the Ages & Stages
Questionnaires: Jamaican version (ASQ-J) and the development of the Ages & Stages
Questionnaires (ASQ-J) system, including focus group and utility studies that led to selected
questionnaire revisions. Normative studies in Jamaica resulted in recruiting and collecting 876
questionnaires. The collected data have been used to examine selected psychometric parameters
of the ASQ-J. In addition to describing the demographic characteristics of the samples, analyses
address test–retest reliability and measures of internal consistency. A comparison of
questionnaire performance by groups of risk and non-risk children is presented. Validity analyses
include descriptions of how the cutoff points were determined and of measures of concurrent
validity.
Introduction and Background. The Ages and Stages Questionnaires (ASQ) are a
series of parent-completed questionnaires designed to accurately identify infants and young
children who are in need of further evaluation and may benefit from early intervention or early
childhood special education services. Because the questionnaires are completed by parents or
caregivers and have flexible administration procedures, they have been easily incorporated into a
variety of health, educational and social service settings in the United States, (Squires, Bricker,
Twombly, & Potter, 2009) as well as a number of other countries including China (Bian, Yao,
Squires, Wei, Chen, & Fang, 2010), Korea, (Heo, Squires, & Yovanoff, 2008), Turkey (Kapci,
Kucuker, & Uslu, 2010) and the Caribbean (Roving Caregiver Program; http://www.ccsi-
18
INSPIRE Jamaica: Screening and Early Intervention System for Children and Families
info.com/).
University of Oregon Project INSPIRE staff recommended using the Ages and Stages
Questionnaires (ASQ) as the in-depth developmental screening tool for the most “at-risk”
children and/or those identified as having concerns based on the current developmental
surveillance method used during well child visits in Jamaican clinics. The flexibility of the ASQ
system should work well given Jamaica’s objective of developing a screening tool that can be
used by parents, primary health care, and early childhood providers in Jamaica. Secondly, while
the ASQ-3 was developed in the United States (US), and the data used to investigate the
psychometric properties and establish cutoff points have been gathered primarily from families
and children who reside in the US, a number of researchers and practitioners from outside of the
US have translated or adapted the tool for their use. The ASQ has been translated into over ten
languages, including Korean, French-Canadian, French, Turkish, Norwegian, Vietnamese,
Chinese, Portuguese, and most recently for Aboriginal families in Australia. The ASQ system
has been extensively studied by researchers in these countries and overwhelmingly has been
found to be valid and easy to translate and adapt for use with different cultures (Bian, Yao,
Squires, Wei, Chen, & Fang, 2010; Kapci, Kucuker, & Uslu, 2010; Heo, Squires, & Yovanoff,
2008; Dionne, Squires, & Leclerc, 2004; Janson & Squires, 2004). Finally, the tool has been
extensively studied and is recommended for use by the American Academy of Pediatrics, (AAP,
2006).
Development of the ASQ-J Translation and adaptation of a measure for use in a different
culture requires several safeguards. As a number of authorities have cautioned, the adoption of a
measure into a different culture requires careful consideration to ensure its appropriate use (e.g., Ball
& Janyst, 2008). Screening or assessment tools that are not culturally appropriate may yield
misleading or negative results and unintended consequences (Musquash & Bova, 2007). Between
August 2012 and November 2012, researchers from the University of Oregon’s Early Intervention
Program collaborated with researchers from the University of West Indies to discuss the
development of an adapted version of the ASQ for use with children and families in Jamaica. In
December 2012, University of Oregon researchers conducted a focus group in Kingston, Jamaica on
the original ASQ with parents, teachers, Jamaican stakeholders, and University of West Indies
research assistants. Focus group participants reported that the overall structure, intent and items on
the ASQ were very appropriate for children and families in Jamaica but suggested that minor
19
INSPIRE Jamaica: Screening and Early Intervention System for Children and Families
changes and adaptations to the tool would allow it to better reflect Jamaican language and culture.
Based on feedback, changes were made to the ASQ questionnaires, and a pilot version of the ASQ-J
was developed.
The Ages and Stages Questionnaires-Jamaica (ASQ-J) is an adaptation of the U.S.
version of the ASQ. The original ASQ system consists of a series of 21 questionnaires that
screen children at specific age intervals, between the ages of 1 to 66 months. Twelve ASQ-J
intervals were developed (from the original 21), and 8 were prioritized for research (i.e., the 6,
12, 18, 24, 30, 36, 54 and 60 month ASQ-J intervals). Each ASQ-J questionnaire contains 30
developmental items that are written in simple, straightforward language. The items are
organized into five areas: Communication, Gross Motor, Fine Motor, Problem Solving, and
Personal-Social. An Overall section addresses general parental concerns. The prioritized
intervals fit into key components of Jamaica’s current system of health and educational systems.
This pilot version includes examples in dialect (Patois) for a number of items in the
communication domain, both in how the question was asked, and examples of correct responses.
In addition, changes were made to item examples that better reflected materials, games, animals,
foods, etc. common in Jamaica (e.g., “mouse” was changed to “ant”; “applesauce” was changed
to “sand”). In addition, the direction “check the circle” when selecting an item response was
changed to “tick the circle”. Items that included units of measurement were changed from the
US system (e.g., inches) to the Jamaican system (e.g., centimeters). The ASQ-J is intended to
be used in the following settings in Jamaica:
Health Clinic Well Child Check-Ups. The ASQ-J will be used a second level screening
tool during well child checkups. During well child checkups, all children are currently
screened at 6, 9, 12, 18, 24 and 36 months using the “11 questions”, included in Jamaica’s
Child Health and Development Passport. The proposed ASQ-J should be administered as
a second level screening when any concern is indicated on the “11 Questions”.
Preschool Programs as part of School Readiness Assessment. The ASQ-J is the parent
component of the School Readiness Assessment that should occur during the end of the
child’s 4/5 school year. In the school setting, the ASQ-J should be completed by the
parent or by the parent and teacher together.
20
INSPIRE Jamaica: Screening and Early Intervention System for Children and Families
Research Objectives and Questions
One of the key objectives of Project INSPIRE was to study the psychometric properties of
the ASQ-J. Research was divided into 2 phases; Phase 1 focused on the utility of the ASQ-J for
Jamaican children and families. Phase 2 focused on descriptive statistics, reliability and validity
studies. Following are research questions for each phase:
1) Phase 1. Research Questions
a. What is the utility (including cultural sensitivity) of the ASQ-J for children,
families and service providers in Jamaica?
b. What modifications need to be made to the ASQ-J?
2) Phase 2. Research Questions
a. What are descriptive statistics (e.g., means, standard deviations) of the ASQ-J for
Jamaican children and families?
b. What is the reliability (test-retest) of the ASQ-J?
i. What is the internal consistency of the ASQ-J?
c. What is the validity of the ASQ-J (e.g., sensitivity, specificity) when compared to
known groups of children with/without disabilities?
Phase 1. Utility Data Collection on the Pilot ASQ-J
In early February 2013, the UWI research team began to collect pilot data on the utility of
the ASQ-J. Collection of utility data continued through mid-July of 2013. Research Assistants
collected ASQ-J’s from parents in Health Clinics and teachers in preschool classrooms. Teachers
and research assistants also completed utility forms indicating the clarity, appropriateness and
meaningfulness of ASQ-J items. Teachers were asked whether they planned to use the tool again
and how they might change it make it better. Results of utility surveys are summarized below. In
May 2013 during a visit to UWI, University of Oregon researchers conducted a second round of
focus groups with parents and teachers and research assistants on the piloted ASQ-J. Participants
in the May focus groups suggested minor formatting changes to the ASQ-J, along with a few
specific suggestions for item changes. These suggestions are listed below:
18 month ASQ GM#5: Change qualifier to “child may or may not hold onto the wall”.
Provide more illustrations across items to assist with low literacy.
Provide additional space for children to copy letters, draw person, etc.
21
INSPIRE Jamaica: Screening and Early Intervention System for Children and Families
60 month ASQ Com#2: Does your child use 4 and 5 word sentences? For example does
your child say, “I want the car”? Provide a Patois example.
60 month ASQ Com #5: What do you do when you get tired? Change to: What do you
do when you get tired at night? (Children have been answering “drink water.”)
Parents and teachers also described the following benefits of the ASQ-J based on their
experiences using the tool. Parents said that the ASQ-J:
Provided useful information
Made you think about things child/baby was doing
Was reassuring to know that child was okay, could do things asked
Brought to light things might not be seen as developmental progress
Highlighted things that could be tried with child or could encourage child to do. Gives
ideas of what to expect at which age.
Was a good way to keep track without doing a big developmental assessment
Domain aspect is very helpful gives idea of where to encourage child e.g., may need
encouragement in gross motor but not in communication.
Teachers made the following comments:
Get to know things you did not know about the child (e.g., medical history, hearing)
Helps parents to know the type of skills their child knows. Learning experience for
parents to know how to monitor their child for certain activities.
Utility Survey Data Collection One hundred ten (110) utility surveys were collected on
the Jamaican version of the Ages and Stages Questionnaires (ASQ-J) across intervals from 2-66
months. The target goal was to collect 80 utility surveys overall. Seventy-eight (78) utility
surveys were collected from parents, 30 from preschool teachers, one from a child’s auntie, and
one from a child’s grandmother. For some of the intervals, such as 24 months, it was more
difficult to recruit parents to complete surveys since these are not the typical times for well child
visits. However, in order to develop a comprehensive set of questionnaires, utility was collected
on all intervals. Eight of the intervals (highlighted below) are “priority intervals”. These intervals
correspond to well child checkups conducted in the Jamaican health care system, and the
proposed aged children will be screened to determine “School Readiness”. The following two
22
INSPIRE Jamaica: Screening and Early Intervention System for Children and Families
tables summarize the number of utility surveys collected by interval (Table 3.1) and responses to
the utility questionnaire (Table 3.2). See Appendix A for INSPIRE Utility Survey.
Table 3.1
Number of utility surveys collected by ASQ-J interval
ASQ-J
intervals
2
mo
6
mo*
10
mo*
12
mo
18
mo*
24
mo*
36
mo*
48
mo
54
mo*
60
mo*
Current
Utility
N
12 10 15 7 16 11 12 3 11 12
Table 3.2
Summary of Responses to Utility Statements*
Total Strongly
Agree Agree No Opinion
Disagree Missing
N N (%)
N (%)
N (%)
N (%)
N
In general, questions
clear and easy to
understand
110 54 (41.9%)
55 (50%) 0 (0%) 1 (.9%) 0 (0%)
Questions
appropriate for child
and family’s culture
110 31 (28.2%) 77 (70%) 1 (.9%) 1 (.9%) 0 (0%)
Completing tool
provided meaningful
Information
110 28 (28.5%) 62 (56.4%) 5 (4.5%) 15
(13.6%)
0 (0%)
Would like to use
this tool again
110 27 (24.8%) 76 (69.7%) 5 (4.6%) 1 (.9%) 1 (.9%)
Question
appropriate for
child’s age
110 30 (27.3%) 59 (53.6%) 18 (16.4%) 3 (2.7%) 0 (0%)
*Response to statements (e.g., In general, were the questions clear and easy to understand?) were
on a 5 point Likert Scale, (Strongly Agree, Agree, No Opinion, Disagree and Strongly Disagree).
In no cases, did participants use the “Strongly Disagree” option, so it is not included in the
summary table.
23
INSPIRE Jamaica: Screening and Early Intervention System for Children and Families
Additional Written Comments:
“Some questions she (the baby) wouldn’t reach yet”
“There are something that I wouldn’t expect baby to do”
“For now, instrument is well designed”
“It was really a privilege, helped me in my studies. Focuses on observation and not
standardized test is better”
How would you change this tool to make it better?
“Give the teachers the kit to work with and ask less questions”
“Separate questions from answers to make it not jumbled”
“Space should be provided for the child’s drawings”
Summary of Utility Findings Utility data on the ASQ-J were quite positive. Almost all—
99.1% of respondents--strongly agreed/agreed that the questions on the ASQ-J were clear and
easy to understand. Ninety-eight percent (98%) of respondents strongly agreed/agreed that
questions were appropriate for their child and family’s culture. Eighty-two (82%) percent of
respondents strongly agreed/agreed that completing the tool provided meaningful information
about the child; 94.5 % of respondents strongly agreed/agreed that they would like to use this
screening tool again. Nearly 81% of respondents strongly agreed/agreed that the questions were
appropriate for their child’s age. Preschool teachers felt that the ASQ-J added valuable
information to the School Readiness Tool and they strongly recommended keeping it for this
purpose.
Based on these findings and anecdotal reports from the research team from the UWI, it
was determined that the ASQ-J appears to be an appropriate and useful child developmental
screening tool for Jamaican populations. Some minor revisions were made to the pilot version of
the ASQ-J formatting and wording. For example, a larger space was provided within the
protocol on the 54 and 60 month ASQ-J intervals for older children to demonstrate their ability to
draw shapes and letters. On the 48 and 60 month intervals, focus group participants reported
problems with a question that asks children to answer the question, “What do you do when you’re
tired?”. Acceptable answers include such phrases as “go to bed”, but a number of children in
Jamaica were answering, “drink water”. The question was modified to, “What do you do at night
24
INSPIRE Jamaica: Screening and Early Intervention System for Children and Families
when you are tired.” Additional responses in Patois were added to a number of intervals. This
“field version” of the ASQ-J was used in Phase 2 data collection.
Phase 2. Psychometric Data Collection on the ASQ-J
The second phase of data collection involved collecting psychometric data on the field
version of the ASQ-J. Questionnaires were completed by parents and caregivers accessing
Jamaican health and educational settings. Teachers of preschool age children were also involved
in completing ASQ-J questionnaires on children older than age 36 months in school settings.
Research staff from the University of West Indies assisted in identifying settings that provided as
well stratified a sample of Jamaican families and children as possible, given financial and
logistical constraints of the project. All parents/caregivers in targeted settings were provided
with information about the child and family risk screening projects and asked if they would be
willing to participate. As part of participation in the research, parents received an ASQ-J
questionnaire from the provider along with a form asking for the child’s demographic
information and a research consent form. The ASQ-J questionnaire was completed either
independently by the parent or with assistance from a research assistant or an educational or
healthcare provider. The completed questionnaires were scored, and the results were shared with
the parent or caregiver. The provider assisted the parent with referrals to community support
services as needed. Procedures ensuring protection of human participants were approved by the
University of West Indies and the University of Oregon institutional review boards and were
followed in all research phases.
The following data are based on 876 questionnaires completed by parents of children
between 1 and 66 months of age. The numbers of questionnaires collected by interval are shown
in Table 3.3.
Population Sample
The data analyses that are contained in this report are based on 876 completed
questionnaires. Each parent or caregiver who completed a questionnaire was asked to complete a
demographic form. See Appendix B for INSPIRE Child/family Demographic Form.
Demographic data included information on the child’s age, gender, parish the child resides,
setting screening took place, who completed screening, number of family household resources
25
INSPIRE Jamaica: Screening and Early Intervention System for Children and Families
(as a proxy of risk), and whether the child was known to have any health risks or known
disabilities. It should be noted that there were 231 missing demographics, so percentages
reported are based on the 645 data for which demographic information was collected. The
demographic data for the population sample are displayed in Tables 3.4-3.7.
Table 3.3
Frequency and Percentage of Data Collected by ASQ-J Interval*
Interval Frequency Percent
2 25 2.9
6* 102 11.6
8 5 .6
10* 106 12.1
12* 92 10.5
18* 113 12.9
24* 94 10.7
30 4 .5
36* 104 11.9
48 18 2.1
54* 114 13.0
60* 99 11.3
Total 876 100.0
* These intervals were prioritized during data collection
Gender of child and parish child resides. As shown in Table 3.4, the gender distribution for the
sample was 53.5% male and 46.5 % female. Table 3.5 contains data on the parish where the
child lived at the time of the data collection. The majority of children lived in Kingston (59.8%),
St. Andrew (23.3%), or St. Catherine (7.9%) Parishes. A small percentage of children (1.5%)
resided in other parishes in Jamaica.
Settings Screenings Took Place Table 3.6 contains data on the setting where the screening took
place. A large percentage (87%) of screenings occurred in health settings, 9.1% occurred in
educational settings, and 3.9% took place in “other” settings. Approximately 92% of these
settings were public and 7.6% were private settings. Over sixty-two percent (62.4 %) of
26
INSPIRE Jamaica: Screening and Early Intervention System for Children and Families
screenings were conducted in urban settings, 32.6% were conducted in “inner” urban settings,
while only 5% of screening occurring in rural settings. See the Limitations of Study section for a
discussion of the imbalance or urban/rural settings and recommendations for future research on
this tool.
Table 3.4
Gender of Children
Frequency Valid Percent
Male 345 53.5
Female 300 46.5
Total 645 100.0
Missing 231
Total N 876
Table 3.5
Parish Where Child Resides
Person Completing Questionnaires Table 3.7 contains data on the person completing the
questionnaires. The majority of individuals completing the questionnaires were mothers (92.1%).
This finding is consistent with feedback from hundreds of screening professionals who report
that mothers are most apt to complete the questionnaires on their children. It should be noted
that the majority of these (87.1%) were completed with the support of a research assistant.
While ASQ-J questionnaires can be completed independently by parents, data collection methods
may have increased the amount of support research assistants provided. In order to determine
Parish Frequency Valid Percent
Kingston 396 59.8
St Andrew 204 30.8
St Catherine 52 7.9
Other 10 1.5
Total 662 100.0
Missing 214
Total 876
27
INSPIRE Jamaica: Screening and Early Intervention System for Children and Families
how much support is required in actual health and educational settings in Jamaica, further
research would need to be conducted.
Table 3.6
Setting Where Screening Took Place
Setting Frequency Valid Percent
Antenatal 18 2.7
Clinic 557 84.1
PATH Home Visit 1 .2
Early Childhood Ed. 34 5.1
Primary School 3 .5
Nursery 23 3.5
Other 26 3.9
Total 662 100.0
Missing 214
Total 876
Table 3.7
Person Completing Questionnaire
Frequency Valid Percent
Mother 610 92.1
Father 20 3.0
Grandmother 15 2.3
Aunt 6 .9
Stepmother 1 .2
Sister 1 .2
Other-family 7 1.1
Non-family 2 .4
Total 662 100.0
Missing 214
Total 876
28
INSPIRE Jamaica: Screening and Early Intervention System for Children and Families
Household resources (child risk). Data on the number of household resources were
collected as a proxy of child risk status. Parents were asked to self-report household resources
from a list of 26 resources (e.g., telephone, car, water heater). A total number of resources was
entered for each family. These data are displayed in Table 3.8. Nearly two percent (1.8%) of
the sample reported having between 1-4 household resources, 19.5% reported having 5-8
resources, 31.6% reported having 9-12 resources; 29.9% reported having 13-16 resources; and
17.1% reported having more than 17 household resources.
Table 3.8
Total Number Household Resources
Cutoff Scores and Percent Identified of Population
Because of the limited number of questionnaires available to establish “Jamaican”
cutoffs, an approach examining the percentage of children identified as “at-risk” was chosen to
determine cutoffs for the ASQ-J. Data were analyzed using the cutoffs from the third edition of
the U.S. version of the ASQ (the ASQ-3), which were established based on a stratified sample of
more than 18,000 children from the U.S. Concurrent validity studies conducted on the ASQ-3
indicated that the best domain cutoffs for optimizing sensitivity and specificity were at 2
standard deviations below the mean.
Using the INSPIRE data set, children were classified as “at-risk” if scores were below
established U.S. cutoffs, or “no-risk” if scores were above established U.S. cutoffs. There are 5
# of Resources Frequency Valid Percent
1-4 12 1.8
5-8 127 19.5
9-12 206 31.6
13-16 195 29.9
17-20 102 15.6
21-24 10 1.5
Total 652 100.0
Missing 224
Total 876
29
INSPIRE Jamaica: Screening and Early Intervention System for Children and Families
developmental areas or “domains” on the ASQ (e.g., communication, fine motor) and scores can
be below cutoff or “at-risk” in 0 to 5 areas. Targets of 12%–16% of children identified in one
developmental area (i.e., one area below the cutoff score) and 2%–7% identified in two or more
areas were adopted as the desired percentages to be identified for further assessment at each age
interval. These figures were based on U.S. Census Bureau and Centers for Disease Control and
Prevention prevalence data related to developmental disabilities in young children as well as
additional resources on worldwide disability prevalence (Cornell University, 2003–2009; U.S.
Census Bureau, 2004).
Table 3.9 contains data on the total number of Jamaican children identified as “at-risk” in
1 or more domains (17.7%), which is slightly higher than the 15.5 % of U.S. children identified
as “at-risk” in studies of the ASQ-3 (the range across intervals was 12-19.2%). Table 3.10
contains data on the total number of Jamaican children identified as “at-risk” in 2 or more
domains across intervals (4.7%) which is lower than the 5.9% of US children identified as “at-
risk” in ASQ-3 research (although still within the range across intervals which was 3.1-7.9 %).
Table 3.11 illustrates the percentage of children identified as “at risk” by number of areas low.
Eighty-two percent (82%) of all screenings conducted had no areas below cutoff. Two children
(.2%) received scores below cutoff in 5 areas on the ASQ.
It is not surprising that U.S. cutoffs may operate quite similarly for Jamaican populations.
Jamaican health and educational expectations are similar to US expectation and standards.
Similar materials (e.g., writing tools, scissors, puzzles) are present in preschool settings in
Jamaica (at least in the urban schools observed), and no materials were considered “culturally
inappropriate” during focus group sessions with Jamaican providers and parents. There will be
settings in Jamaica where materials are limited and children have not had opportunity to try
items on the ASQ-J, also common in the U.S. as well. Instructions for administering the tool
always recommend making materials available for children so they can practice skills prior to
administering the tool. Many times the process of screening provides opportunities for parents
and providers to think about how to make these new experiences available to young children.
Table 3.12 contains data on the percent of children identified as “at-risk” by the number of
reported household resources. Percentages range from a low of 15% to a high of 25% in
households with less than 4 reported resources. It is difficult to draw any conclusions based on
these data since there are only 12 children in the category reporting less than 4 household
30
INSPIRE Jamaica: Screening and Early Intervention System for Children and Families
resources. Demographic data, including the number of household resources, were missing in 224
cases. The overall percent of children identified as “at-risk” in 1 area on the ASQ-J, as reported
earlier, was 17.7%.
Table 3.9
Percent of Children Identified as “At-Risk” in 1 or More Domains Using U.S. Cutoffs
Table 3.10
Percent of Children Identified as “At-Risk” in 2 or More Domains using US Cutoffs
Reliability Studies
Reliability studies completed on the ASQ-3 include test–retest reliability and internal
consistency of ASQ-3 items. Internal consistency was examined using correlational analyses
and the Cronbach coefficient alpha (Cronbach, 1951). Each of these analyses is presented next.
Test–Retest Reliability. Test–retest reliability is designed to help determine the stability of test
outcomes over time. Test–retest reliability of the ASQ-J was examined by comparing two
questionnaires completed by the same caregiver at a 2-3 week time interval.
Child status
Frequency
Jamaica
Total %
Jamaica
Frequency
US data
Total %
US data
No-risk
(above cutoff)
721 82.3 15, 687 84.5%
At-risk (below
cutoff)
155 17.7 2885 15.5%
Total 876 100.0 18, 572
Child status
Frequency
Jamaica
Percent
Jamaica
Frequency
US data
Percent
US data
No-risk
(above cutoff)
835 95.3 17,478 94.1%
At-risk (below
cutoff)
41 4.7 1094 5.9%
Total 876 100.0 18, 572 100%
31
INSPIRE Jamaica: Screening and Early Intervention System for Children and Families
Table 3.11
Percent of Children Identified by Number of Areas below Cutoff “At risk”
Table 3.12
Percent Identified as “At-Risk” in 1 Area on ASQ-J by Number of Household Resources
That is, parents were asked to complete the same questionnaire interval for their child twice
within a 2-3 week time period between completions. Questionnaires completed by 45 parents
were included in this analysis. Parents were blind to the results of the first questionnaire when
they completed the second one. The two questionnaires completed by parents were then
compared for agreement on classifications (i.e., screened or not screened). As shown in Table
3.13 the percent agreement for the 45 parents was 86.7 %. Intraclass correlations, as shown in
# of areas below cutoff
“at-risk” Frequency Percent
0 areas 721 82.3
1 area 114 13.0
2 areas 30 3.4
3 areas 6 .7
4 areas 3 .3
5 areas 2 .2
Total 876 100.0
# of Resources N % Identified
“At-Risk”
1-4 12 25.0
5-8 127 15.7
9-12 206 15.0
13-16 195 15.4
Over 17 112 15.2
Missing 224 24.1
Total 876 17.7
32
INSPIRE Jamaica: Screening and Early Intervention System for Children and Families
Table 3.14, ranged from a low of .43 to a high of .71 indicating robust test–retest reliability
across ASQ developmental areas.
Table 3.13
Test-Retest Agreement
Table 3.14
Intraclass Agreement
Internal Consistency The internal consistency of the questionnaires was addressed by
examining the relationship between developmental area and overall scores. Correlational
analyses and Cronbach coefficient alpha (Cronbach, 1951) were calculated. Pearson product
moment correlation coefficients were calculated for developmental area scores with an overall
ASQ-J score for questionnaire age intervals. Correlations were computed on the 8 prioritized
ASQ-J intervals, which had sufficient data to analyze. As shown in Table 3.15, the correlations
by developmental area and overall ASQ-J score are consistent and generally ranged from .59 to
.86. The major exception is the Gross Motor area, in which 5 correlations were below .57. All
correlations are significant at p < .01. These findings suggest moderate to strong internal
consistency between developmental areas and total test score.
Time 1 & Time 2
Classification (Same
or Different) Frequency Percent
Different 6 13.3
Same 39 86.7
Total 45 100.0
Communication 0.673
Gross Motor 0.614
Fine Motor 0.718
Problem Solving 0.434
Personal-Social 0.642
33
INSPIRE Jamaica: Screening and Early Intervention System for Children and Families
Table 3.15
Correlations between Area and Overall Score
Table 3.16 contains the correlations between developmental area scores that have been
collapsed across the 8 questionnaire age intervals. Again, all correlations were significant,
suggesting congruence between developmental areas as well as between developmental areas
and overall ASQ-J scores.
Table 3.16
Correlations between Area Scores Collapsing Across Questionnaires
Age
Interval n Communication
Gross
Motor
Fine
Motor
Problem
Solving
Personal-
Social
6 102 0.59 0.73 0.71 0.78 0.71
10 106 0.62 0.53 0.71 0.72 0.70
12 92 0.63 0.41 0.64 0.79 0.76
18 113 0.68 0.44 0.68 0.73 0.61
24 94 0.71 0.44 0.82 0.73 0.77
36 104 0.59 0.57 0.79 0.55 0.67
54 114 0.76 0.67 0.73 0.86 0.69
60 99 0.69 0.74 0.80 0.75 0.67
Area Communication
Gross
Motor
Fine
Motor
Problem
Solving
Personal-
Social
Communication
Gross Motor 0.32
Fine Motor 0.25 0.24
Problem
Solving 0.20 0.24 0.48
Personal-Social 0.25 0.21 0.28 0.37
Overall 0.63 0.56 0.70 0.72 0.65
Note: All correlations are significant at p < 0.01
34
INSPIRE Jamaica: Screening and Early Intervention System for Children and Families
The reliability of the ASQ-J questionnaires has been studied by examining the internal
consistency and test–retest reliability of the questionnaires. Internal consistency analyses have
indicated strong relationships across items and within areas on the questionnaires. The
questionnaires also achieved substantial test–retest reliability. Parents’ evaluations of their
children using the questionnaires were consistent over time.
Validity Studies
Financial and logistical constraints of the grant made the gathering of validity data on the
ASQ-J impossible.
Summary of Findings on the ASQ-J
Initial utility and psychometric studies on the ASQ-J are promising. Both providers and
parents found that the ASQ-J to be an appropriate and useful child developmental screening tool
for Jamaican populations. The current U.S. cutoffs seem to be operating similarly for Jamaican
populations and identifying a similar percentage of children “at risk” and in need of further
evaluation. Research indicates good reliability with the ASQ-J, both across users and internally
in the test construction.
Limitations and recommendations. There are limitations to this study. The vast majority
of data collected were from 3 parishes in Jamaica (i.e., Kingston, St. Andrews and St.
Catherine’s) and the majority (95%) in urban or inner urban settings. More research on the
appropriateness of this tool across parishes and in rural settings would be recommended. Most
likely access to some materials required to demonstrate skills (e.g., scissors, crayons) may be
limited in rural community and outcomes may differ. It will be important for administrators to
consider how they can make screening materials available to children and parents prior to
completing the ASQ-J. How much support parents will need to complete questionnaires is
unknown, given research assistants supported most parents who completed questionnaires.
Again, screening administrators will need to consider that parents may need some support while
completing the ASQ-J (e.g., reading, interpreting items). The number of questionnaires, and the
stratification of the data are not sufficient to establish specific “Jamaican” cutoffs as yet, so
continuing to gather data on this tool would be recommended.
35
INSPIRE Jamaica: Screening and Early Intervention System for Children and Families
Part II
SWYC Technical Report
This section offers a range of technical information about the Survey of Wellbeing of
Young Children (SWYC) and the development of the SWYC-Jamaica system, including focus
group and utility studies that led to selected questionnaire revisions. Normative studies in
Jamaica resulted in recruiting and collecting 647 questionnaires. The collected data have been
used to examine selected psychometric parameters of the SWYC. In addition to describing the
demographic characteristics of the samples, analyses included in this report addresses test–retest
reliability and measures of internal consistency. Validity analyses include descriptions of how
the cutoff points were determined and of measures of concurrent validity.
Introduction and Background. The Survey of Well-being of Young Children (SWYC) is a
freely-available, comprehensive screening instrument for children under 5 years of age. There
are four segments of the original SWYC, though the Family Context portion was omitted for
Project INSPIRE, as this area would be screened thoroughly using the Jamaican Family Support
Screening Tool. The three SWYC components currently being studied through the INSPIRE
project are: (1) the Milestones: a short developmental component with questions about
cognitive, language and motor development; (2) the BPSC and PPSC: questions focused on
children’s social-emotional and behavioral development; and (3) the POS: questions focused on
screening for autism.
Development of the SWYC Translation and adaptation of a measure for use in a different
culture requires several safeguards. As a number of authorities have cautioned, the adoption of a
measure into a different culture requires careful consideration to ensure its appropriate use (e.g.,
Ball & Janyst, 2008). Screening or assessment tools that are not culturally appropriate may yield
misleading or negative results and unintended consequences (Musquash & Bova, 2007). In
December 2012, University of Oregon researchers conducted a focus group in Kingston, Jamaica
on the SWYC with parents, teachers, Jamaican stakeholders, and University of West Indies
(UWI) research assistants. Focus group participants reported that the overall structure, intent and
items on the SWYC were very appropriate for children and families in Jamaica and had no
significant suggestions for changes.
The SWYC is intended to be used as the second-level screening tool during well child
checkups, when any behavioral concerns were identified. During well child checkups, all
36
INSPIRE Jamaica: Screening and Early Intervention System for Children and Families
children are currently screened at 6, 9, 18, 24 and 36 months using the “11 questions”, included in
Jamaica’s Child Health and Development Passport. The SWYC would be administered as a
second level screening when any behavioral concern was indicated on the “11 Questions”.
Research Objectives and Questions
One of the key objectives of Project INSPIRE was to study the psychometric properties of
the SWYC. Research was divided into 2 phases; Phase 1 focused on the utility of the SWYC for
Jamaican children and families. Phase 2 focused on descriptive statistics, reliability and validity
studies. Following are research questions for each phase:
1) Phase 1. Research Questions
a. What is the utility (including cultural sensitivity) of the SWYC for children,
families and service providers in Jamaica?
b. What modifications need to be made to the SWYC?
2) Phase 2. Research Questions
a. What are descriptive statistics (e.g., means, standard deviations) of the SWYC for
Jamaican children and families?
b. What is the reliability (test-retest) of the SWYC?
c. What is the validity of the SWYC (e.g., sensitivity, specificity) when compared to
known groups of children with/without disabilities?
Phase 1. Utility Data Collection on the Pilot SWYC
In early February 2013, the UWI research team began to collect pilot data on the utility of
the SWYC. Collection of utility data continued through mid-July of 2013. Research Assistants
interviewed parents in Health Clinics to complete the SWYC. Parents also completed utility
surveys indicating the clarity, appropriateness and meaningfulness of SWYC items. Results of
utility surveys are summarized below. In May 2013 during a visit to UWI, University of Oregon
researchers conducted a second round of focus groups with parents and research assistants on the
piloted SWYC. Participants in the May focus groups were satisfied with the tool and did not
suggest any changes.
Utility Survey Data Collection Seventy-nine (79) utility surveys were collected on the
Survey of Wellbeing of Young Children (SWYC) across intervals from 2-36 months. Five of the
37
INSPIRE Jamaica: Screening and Early Intervention System for Children and Families
intervals (i.e., 6, 19, 18, 24, 36) are “priority intervals”. These intervals correspond to well child
checkups conducted in the Jamaican health care system; the additional intervals were more
difficult to gather utility data on since few children are seen at clinics at these times. Data were
collected across SWYC intervals, however, to have information on a comprehensive set of
questionnaires. The following tables summarize the number of utility surveys collected by
interval (Table 3.17) and responses to the utility questionnaire (Table 3.18). Seventy-five percent
(75%) of parents reported that the SWYC took less than 10 minutes to complete, (15 parents did
not respond to this question).
Table 3.17
Number of Utility Surveys Collected by SWYC Interval
Summary of Utility Findings Utility data on the SWYC collected thus far, while limited, are
positive. One hundred percent (100%) of respondents strongly agreed/agreed that the questions
on the SWYC clear and easy to understand. One hundred percent (100%) of respondents strongly
agreed/agreed that questions are appropriate for their child and family’s culture. A little over
ninety-three percent (93.4%) of respondents strongly agreed/agreed that completing the tool
provided meaningful information about the child while 93.6% of respondents strongly
agreed/agreed that they would like to use this screening tool again. More than 97% percent of
respondents strongly agreed/agreed that the questions were appropriate for their child’s age.
When asked how to change the tool to make it better, 93.7% said they had no changes. Based on
current findings and anecdotal reports from the research team from the University of West Indies,
the SWYC appears to be an appropriate and useful child developmental/ behavioral screening tool
for Jamaican populations.
SWYC
intervals
2
month
6
month
9
month
12
month
18
month
24
month
36
month
# of completed
utility surveys
2
12
13
7
17
11
13
38
INSPIRE Jamaica: Screening and Early Intervention System for Children and Families
Table 3.18
SWYC: Summary of Responses to Utility Statements*
Total Strongly
Agree Agree No Opinion Disagree Strongly
Disagree Missing
N N (%)
N (%)
N (%)
N (%)
N (%)
N (%)
In general,
questions clear
and easy to
understand
79 39 (49.4%)
40 (50.6%) 0 (0%) 0 (0%) 0 (0%) 0 (0%)
Questions
appropriate for
child and
family’s
culture
79 31 (39.2%) 46 (58.2%) 1 (1.3%) 1 (1.3%) 0 (0%) 0 (0%)
Completing
tool provided
meaningful
Information
79 18 (22.8%) 40 (50.6%) 9 (11.4%) 12 (15.2%) 0 (0%) 2 (6.7%)
Would like to
use this tool
again
79 25 (31.6%) 49 (62.0%) 3 (3.8%) 2 (2.5%) 0 (0%) 2 (6.7%)
Question
appropriate for
child’s age
79 22 (27.8%) 54 (68.4%) 1 (1.3%) 1 (1.3%) 0 (0%) 1 (1.3%)
*Response to statements (e.g., In general, were the questions clear and easy to understand?) were
on a 5 point Likert Scale, (Strongly Agree, Agree, No Opinion, Disagree and Strongly Disagree.
Phase 2. Psychometric Data Collection on the SWYC
The second phase of data collection involved collecting psychometric data on the field
version of the SWYC. Questionnaires were completed by parents and caregivers accessing
Jamaican health services. Research staff from the University of West Indies assisted in
identifying settings that provided as well stratified a sample of Jamaican families and children as
possible, given financial and logistical constraints of the project. All parents/caregivers in
targeted settings were provided with information about the child and family risk screening
projects and asked if they would be willing to participate. As part of participation in the research,
39
INSPIRE Jamaica: Screening and Early Intervention System for Children and Families
parents received an SWYC questionnaire along with a form asking for the child’s demographic
information and a research consent form. The SWYC questionnaire was completed either
independently by the parent or with assistance from a research assistant or a healthcare provider.
The completed questionnaires were scored, and the results were shared with the parent or
caregiver. The provider assisted the parent with referrals to community support services as
needed. Procedures ensuring protection of human participants were approved by the University
of West Indies and the University of Oregon institutional review boards and were followed in all
research phases.
The following SWYC data are based on 647 questionnaires completed by parents of
children between 1 and 36 months of age. The numbers of questionnaires collected by interval
are shown in Table 3.19.
Table 3.19
Frequency and Percentage of Data Collected by SWYC Interval
Population Sample
Each parent or caregiver who completed a questionnaire was asked to complete a
demographic form. Demographic data included information on the child’s age, gender, parish the
child resides, setting screening took place, who completed screening, number of family
household resources (as a proxy of risk), and whether the child was known to have any health
risks or known disabilities. It should be noted that there were some missing demographics, so
Interval Frequency Percent
2 23 3.6
6* 106 16.4
9* 106 16.4
12 102 15.8
18* 113 17.5
24* 92 14.2
36* 105 16.2
Total 647 100
* These intervals were prioritized during data collection
40
INSPIRE Jamaica: Screening and Early Intervention System for Children and Families
percentages reported are valid percentages, and based on the number of data for which
demographic information was collected. Demographic data for the population sample are
displayed in Tables 3.20-3.23.
Gender of child and parish child resides. As shown in Table 3.20, the gender distribution
for the sample was 54% male and 45.5% female. Table 21 contains data on the parish where the
child lived at the time of the data collection. The majority of children lived in Kingston (61.1%),
St. Andrew (30.9%), or St. Catherine (6.6%) parishes. A small percentage of children (1.5%)
resided in other parishes in Jamaica.
Table 3.20
Gender of Children
Table 3.21
Parish Where Child Resides
Frequency Valid Percent
Male 330 54
Female 276 45.5
Total 606 100
Missing 41
Total N 647
Parish Frequency
Valid
Percent
Kingston 380 61.1
St Andrew 192 30.9
St Catherine 41 6.6
Other 9 1.5
Total 622 100
Missing 25
Total 647
41
INSPIRE Jamaica: Screening and Early Intervention System for Children and Families
Settings Screenings Took Place Table 3.22 contains data on the setting where the
screening took place. A large percentage (92.8%) of screenings occurred in health settings,
primarily in health clinic settings during well child checkups. Approximately ninety-four
percent (94.2 %) of these settings were public and 5.5 % were private settings. Almost sixty
percent (59.6%) of screenings were conducted in urban settings, 34.7% were conducted in
“inner” urban settings, while only 5.6 % of screening occurring in rural settings. See the
Limitations of Study section for a discussion of the imbalance or urban/rural settings and
recommendations for future research on this tool.
Table 3.22
Setting Where Screening Took Place
Person Completing Questionnaires Table 3.23 contains data on the person completing the
questionnaires. The majority of individuals completing the questionnaires were mothers (93.2%).
This finding is consistent with feedback from hundreds of screening professionals who report
that mothers are most apt to complete the questionnaires on their children. It should be noted
that the majority of these (86.2%) were completed with the support of a research assistant.
While SWYC questionnaires can be completed independently by parents, data collection
methods may have increased the amount of support research assistants provided. In order to
Setting Frequency Valid Percent
Antenatal 18 2.9
Clinic 558 89.7
PATH Home Visit 1 .2
Early Childhood
Educational
3 .5
Primary School 0 0
Nursery 19 3.1
Other 23 3.7
Total 622 100
Missing 25
Total 647
42
INSPIRE Jamaica: Screening and Early Intervention System for Children and Families
determine how much support is required in actual health and educational settings in Jamaica,
further research would need to be conducted.
Table 3.23
Person Completing Questionnaire
Household resources (child risk). Data on the number of household resources was
collected as a proxy of child risk status. Parents were asked to self-report household resources
from a list of 26 resources (e.g., telephone, car, water heater). A total number of resources was
entered for each family. These data are displayed in Table 24. A little over two percent (2.3%)
of the sample reported having between 1-4 household resources, 19.4% reported having 5-8
resources, 31.6 having 9-12 resources, 30.8% having 13-16 resources, 14.7% having 17-20
resources, and 1.3% having 21-24 resources.
Cutoff Scores and Percent Identified of Population
Because of the limited number of questionnaires available to establish “Jamaican”
cutoffs, an approach examining the percentage of children identified as “at-risk” was chosen to
determine cutoffs for the SWYC.
Using the INSPIRE data set, children were classified as “at-risk” if scores were below
established U.S. cutoffs, or “no-risk” if scores were above established U.S. cutoffs. At first, three
Frequency
Valid
Percent
Mother 580 93.2
Father 16 2.6
Grandmother 14 2.3
Aunt 5 .8
Stepmother 0 0
Sister 1 .2
Other-family 5 .8
Non-family 1 .2
Total 622 100
Missing 25
Total 647
43
INSPIRE Jamaica: Screening and Early Intervention System for Children and Families
SWYC component were examined including, (1) the Milestones developmental screen (2) the
BPSC and PPSC social-emotional/behavioral screens and (3) the POS autism screen. Each
section of the SWYC was examined separately.
Table 3.24
Total Number Household
Targets of 12%–16% of children identified in the Milestones developmental segment of
the SWYC were adopted as the desired percentages to be identified for further assessment at
each age interval. These figures were based on U.S. Census Bureau and Centers for Disease
Control and Prevention prevalence data related to developmental disabilities in young children as
well as additional resources on worldwide disability prevalence (Cornell University, 2003–2009;
U.S. Census Bureau, 2004).
Targets of 16%-20% of children identified in the social-emotional components of the
SWYC (i.e., the BPSC, PPSC, POSI) were adopted as the desired percentages to be identified for
further assessment at each age interval. These figures were based on U.S. Census Bureau and
Centers for Disease Control and Prevention prevalence data related to social emotional
disabilities in young children as well as additional resources on worldwide disability prevalence
(Cornell University, 2003–2009; U.S. Census Bureau, 2004).
Table 3.25 contains data on the total number of Jamaican children identified as “at-risk”
on the Milestones developmental screening tool. For intervals 2-12 months, 6.5% of children
# of Resources Frequency Valid Percent
1-4 14 2.3
5-8 119 19.4
9-12 194 31.6
13-16 189 30.8
17-20 90 14.7
21-24 8 1.3
Total 614 100
Missing 33
Total 647
44
INSPIRE Jamaica: Screening and Early Intervention System for Children and Families
were identified as “at-risk” which is low compared to the target of identifying 12-16% of
children in one developmental area. For intervals 18-36 months, 13.3% of children are identified
as “at-risk”, which is within the target identification range. While these findings are promising,
during a consultation with researchers from Tufts University (the developers of the
SWYC), it was learned that the Milestones section of the SWYC was substantially revised
from the version available December 2012. For this reason, the data collected on the
Milestones component of the SWYC in this study are not valid. Other sections of the
SWYC have not been revised at this time and the data are valid for these sections.
Table 3.26 contains data on the percentage of Jamaican children identified as “at-risk” on
the BPSC, which is the social-emotional screening component of the SWYC, for children 2-12
months. Using the recommended cutoff of 3 or more concerns in any one of 3 sections, a high
percentage of children (46.4%) are identified. Because of this finding, an analysis was also run
on the percent of children identified as “at risk” using 4 or more concerns. This analysis resulted
in 30.5% of children being identified as “at-risk”. Even with an adjusted cutoff, this percentage
is high when compared to target identification rates of 16-20%.
Table 3.27 contains data on the percentage of Jamaican children identified as “at-risk” on
the PPSC, which is the social-emotional screening component of the SWYC for children 18-36
months. Using the recommended U.S. cutoff of 9 or more concerns, 30.4% of children are
identified as “at-risk”. This identification rate is high for the recommended target identification
range.
Table 3.28 contains data on the percentage of Jamaican children identified as “at-risk” on
the POSI, which is the autism screening component of the SWYC for children older than 18
months. Using the recommended U.S. cutoff, 26.5 % of children are identified as “at-risk” for
autism. Similar to the PPSC and BPSC, this percentage is high when compared to target
identification rates.
45
INSPIRE Jamaica: Screening and Early Intervention System for Children and Families
Table 3.25
Percent Identified on Milestones Component of SWYC, 2-12 & 18-36 Months
Table 3.26
Percent Identified on BPSC Using 3 or 4 Concerns as Cutoff*
Table 3.27
Percent Identified on PPSC
Classification
Frequency
(2-12)
Percent
(2-12)
Frequency
(18-36)
Valid Percent
(18-36)
Typical 287 93.5 242 86.7
“At-Risk” 20 6.5 37 13.3
Total 338 100 279 100
Missing 0 16
Classification
Frequency
(3 concerns)
Percent
(3 concerns)
Frequency
(4 concerns)
Percent
(4 concerns)
Typical 181 53.6 235 69.5
“At-Risk” 157 46.4 103 30.5
Total 338 100 338 100
*N=338 for intervals 2-12 months
Classification Frequency Percent
Typical 215 69.6
“At-Risk” 94 30.4
Total 309 100.0
*N=309 for intervals 18-36 Months
46
INSPIRE Jamaica: Screening and Early Intervention System for Children and Families
Table 3.28
Percent Identified on POSI
Reliability Studies
Reliability studies completed on the ASQ-3 include test–retest reliability and internal
consistency of ASQ-3 items. Internal consistency was examined using correlational analyses
and the Cronbach coefficient alpha (Cronbach, 1951). Each of these analyses is presented next.
Test–Retest Reliability. Test–retest reliability is designed to help determine the stability
of test outcomes over time. Test–retest reliability of the SWYC was examined by comparing two
questionnaires completed by the same caregiver at a 2-3 week time interval. That is, parents were
asked to complete the same questionnaire interval for their child twice within a 2-3 week time
period between completions. Parents were blind to the results of the first questionnaire when
they completed the second one. The two questionnaires completed by parents were then
compared for agreement on classifications (i.e., typical or “at-risk”). Test-Retest was analyzed
separately for each component of the SWYC (i.e., the BPSC, PPSC and POSI).
As shown in Table 3.29 the percent agreement for the BPSC component of the SWYC
(N=27) was 81.5 %, indicating strong test-retest agreement (intraclass correlation .86). Test-
retest agreement for the PPSC (N=17) was 82.4%, indicating strong test-retest agreement
(intraclass correlation .86). Test-retest agreement for the POSI (N=17) was 29.4%, which is
considered poor agreement (intraclass correlation of -2.1). It is difficult to draw conclusions on
these data given the low N, particularly for the analysis of the PPSC and the POSI. It is
recommended to continue to gather data on these tools.
Classification Frequency Valid Percent
Typical 227 73.5
“At-Risk” 82 26.5
Total 309 100.0
N=309 for 18-36 Month Intervals
47
INSPIRE Jamaica: Screening and Early Intervention System for Children and Families
Table 3.29
Test-Retest Agreement on Classification (typical/”at-risk”) on BPSC, PPSC, and POSI
Components of the SWYC
The reliability of the SWYC questionnaires has been studied by examining test–retest
reliability of the questionnaires. The questionnaires achieved substantial test–retest reliability for
the BPSC and PPSC social-emotional screening components of the SWYC. However, the POSI
component of the questionnaires had unacceptable test-retest reliability. These results need to be
interpreted with caution, however, given the small number of participants in test-retest analysis.
Validity Studies
Financial and logistical constraints of the grant made the gathering of validity data on the
SWYC impossible.
Summary
Initial utility studies on the SWYC are very promising. Both providers and parents found
that the SWYC to be an appropriate and useful social-emotional/behavioral screening tool for
Jamaican populations. Findings on the psychometric qualities of the SWYC have varying results
based on each component studied. The Milestones section of the SWYC has been substantially
revised from the version available December 2012 and data collected in this research are not
valid. The other components of the SWYC (i.e., the BPSC, PPSC and POSI) have not been
revised at this time and these components are the sections of the SWYC that provide information
on the child’s social-emotional/behavioral development (the targeted purpose for use of this tool
in Jamaica). Identification rates for the social-emotional screening components of the SWYC
(the BPSC for intervals 2-12 months and the PPSC for intervals 18 months and older) were
Time 1 & Time 2
Classification
BPSC
N (%)
PPSC
N (%)
POSI
N (%)
Same 22 (81.5%) 14 (82.4%) 5 (29.4%)
Different 5 (18.5%) 3 (17.6%) 12 (70.6%)
Total 22 (100%) 17 (100%) 17 (100%)
48
INSPIRE Jamaica: Screening and Early Intervention System for Children and Families
analyzed. The BPSC identified a very high percentage of Jamaican children as “at-risk” (46.4%)
using recommended U.S. cutoffs. Adjusting the cutoffs (i.e., using 4 or more concerns as “at-
risk”) resulted in 30.5% of Jamaican children being identified. This identification rate is still
high. The PPSC identified 30.4% of children as “at-risk”. This identification rate is high. The
POSI, the autism screening component of the SWYC for children18 months and older, identified
26.8% of children as “at-risk” for autism. Similar to the BPSC and PPSC, this percentage is high
when compared to target identification rates.
Test-retest reliability on the BPSC and PPSC were high (81.5 and 82.4, respectively),
while test-retest reliability on the POSI was poor (29.4). Reliability data are difficult to draw
conclusions from, given the low number of participants included in the analysis, which ranged
from 27 participants (for the BPSC) to 17 participants for the PPSC and POSI analysis. These
analyses should be repeated with a larger data sample.
Limitations and recommendations. There are limitations to this study. The vast majority
of data collected were from 3 parishes in Jamaica (i.e., Kingston, St. Andrews and St.
Catherine’s) and the majority (95%) in urban or inner urban settings. More research on the
appropriateness of this tool across parishes and in rural settings would be recommended. How
much support parents will need to complete questionnaires is unknown, given research assistants
supported most parents who completed questionnaires. Again, screening administrators will need
to consider that parents may need some support while completing the SWYC (e.g., reading,
interpreting items).
Because of the high percentage rates of children being identified as “at-risk” on the
BPSC, PPSC and POSI, screening administrators need to be prepared for a potentially large
percentage of screenings receiving a “red flag” that requires personnel to interpret screening
information and appropriate follow-up necessary (if any). It is difficult to know if these high
identification rates will persist as the tool is implemented in Jamaica and more data are collected.
It is possible that children in Jamaica have a higher number of social-emotional/behavioral
concerns than the estimated prevalence rates of these delays. It is also possible that, because
these components of the SWYC have only undergone preliminary studies, the tool may require
revisions to test items and/or recommended cutoff scores to bring down identification rates.
Because of the importance of screening for social-emotional and behavioral delays it is
recommended to continue using the SWYC as a second level screening when a parent indicates a
49
INSPIRE Jamaica: Screening and Early Intervention System for Children and Families
concern about a child’s behavior. It is also recommended that Jamaican screening system
developers keep up to date with changes to the SWYC. It may be important to update this tool
as additional research is conducted both in Jamaica and the U.S. For current protocols and
research on the SWYC, please go to: http://www.theswyc.org/ In addition, continuing to gather
psychometric data on this tool, including identification rates, reliability and validity data, is
highly recommended.
50
INSPIRE Jamaica: Screening and Early Intervention System for Children and Families
Chapter 4: School Readiness Screening
Introduction and Tool Development
The Jamaica School Readiness Screening was designed to screen children for child
development and readiness for primary school in order determine whether additional
developmental evaluation is necessary, and to assist in curriculum-planning to support children’s
readiness for primary school. Preschool teachers are responsible for completion of the School
Readiness Screening with all children in four-year-old preschool classrooms during Easter term.
Teachers may complete the tool independently or in collaboration with a child’s parents or main
caregivers. The School Readiness tool includes 3 component parts: 1) Part 1: Ages and Stages
Questionnaires-Jamaica (ASQ-J) (Squires et al., 2009), the child development screening tool; 2)
Part 2: Child Behavior Rating Scale; and 3) Part 3: Approaches to Learning/Social-Emotional,
Literacy, and Numeracy Skills. Parts 2 and 3 are described briefly below. Part 1: The ASQ-
Jamaica (ASQ-J) is presented in Chapter 3 of the Technical Report. The ASQ-J is intended to
be completed by the parent with support from the child’s teacher as necessary, while the CBRS
and the School Readiness Skills are completed by the child’s preschool teacher.
Jamaica School Readiness Part 2: The Child Behavior Rating Scale (Bronson,
Goodson, Layzer, & Love, 1990).
The Child Behavior Rating Scale (CBRS) is a seventeen-item survey that assesses a child’s self-
regulatory skills, behaviour with other adults and children in a classroom setting, and social-
emotional development. The CBRS was selected as a component of the School Readiness tool
because self-regulatory and social emotional skills at kindergarten entry have been demonstrated
to be strong predictors of later school success. The CBRS has been used widely in research
studies and has demonstrated strong predictive validity with reading and math achievement in
elementary grades (Sektnan, McClelland, Acock, & Morrison, 2010). It also has been validated
in a wide range of cultural contexts.
Jamaica School Readiness Part 3: School Readiness Skills: Approaches to
Learning/Social Emotional, Early Literacy and Early Numeracy
The School Readiness Skills (SRS): Approaches to Learning/Social Emotional, Early
Literacy and Early Numeracy comprise Part of 3 of the School Readiness Screening. The
51
INSPIRE Jamaica: Screening and Early Intervention System for Children and Families
purpose of this component is to identify areas in which children need to develop competence in
order to be successful in primary school. The School Readiness Skills were developed in 2012
following a literature review of existing school readiness tools, other assessment measures and
existing definitions of school readiness to identify areas most predictive of school readiness;
researchers also looked at the Jamaica Early Childhood Curriculum. This process of review
guided the development and selection of items. The School Readiness Skills were then adapted
in 2013 to incorporate feedback from focus groups with teachers, parents and other key early
childhood stakeholders in Jamaica. The School Readiness Skills address skills universally
identified as important for school readiness and are completed by teachers through daily
observations of students in the classroom setting. Results from the School Readiness Skills may
guide teachers in preparing children for entry into primary school.
Development of the School Readiness Screening. In order to develop the School
Readiness Screening, the University of Oregon research team completed a literature review of
existing school readiness tools and of definitions of school readiness to identify areas most
predictive of school readiness. The Jamaica Early Childhood Curriculum was also reviewed and
consulted. In addition to the child development screening component (ASQ-J), the research team
identified behavior, readiness to learn, social emotional, early literacy and early numeracy as
important skills to include on a school readiness tool and skills with most predictive validity for
later school success. A draft version of the School Readiness screening tool was developed and
included three parts: 1) the ASQ-J child development screening, 2) the Child Behavior Rating
Scale (CBRS), and 3) the School Readiness Skills (SRS). Input on the draft version of all three
components of the School Readiness Screening was gathered from stakeholders during a focus
group. Based on the focus group feedback, items were adapted and/or revised and clarifying
examples included, resulting in the pilot version used for initial data collection.
Research Objectives and Questions
One of the key objectives of Project INSPIRE was to study the psychometric properties of
the CBRS and SRS. Research was divided into 2 phases. Phase 1 focused on the utility of the
CBRS and SRS for Jamaican children and families; Phase 2 focused on psychometric properties
including descriptive statistics, reliability and validity studies. The goal of Phase 2 was to
examine the psychometric qualities of the tools as well as develop cutoff scores for identifying
52
INSPIRE Jamaica: Screening and Early Intervention System for Children and Families
children who need additional support to be ready to enter the primary school setting. Following
are research questions for each phase:
1) Phase 1. Research Questions
a. What is the utility (including cultural sensitivity) of the CBRS and School
Readiness Skills for children, families and teachers in Jamaica?
b. What modifications need to be made to the CBRS and School Readiness Skills?
2) Phase 2. Research Questions
a. What are descriptive statistics (e.g., means, standard deviations) of the CBRS and
School Readiness Skills for Jamaican children and families?
b. What is the reliability (test-retest) of the CBRS and School Readiness Skills?
i. What is the inter-observer reliability for the Early Literacy and Early
Numeracy areas of the School Readiness Skills?
ii. What is the internal consistency of the School Readiness Skills?
c. What is the validity of the CBRS and School Readiness Skills? (e.g., sensitivity,
specificity) when compared to known groups of children with/without
disabilities?
3) Phase 1. Utility Data Collection on the Pilot School Readiness Screening,
Parts 2 and 3
4) During the Pilot Phase, preschool teachers completed Part 2: CBRS and Part 3: School
Readiness Skills of the School Readiness Screening with children in their classrooms.
They completed Part 1: ASQ-J in collaboration with parents and main caregivers.
Teachers then provided utility feedback on the clarity, appropriateness and
meaningfulness of items on all three parts of the tool School Readiness Screening. A copy
of the utility survey can be found in Appendix A at the end of this technical report.
5) The results from Part 1: ASQ-J are contained in Chapter 3 of the Technical Report. The
utility data for Part 2: the CBRS (N = 39) and Part 3: the School Readiness Skills (N = 49)
are presented in Tables 4.1 and 4.2 below.
6) Thirty-nine utility surveys were collected on the CBRS. The following table summarizes
CBRS tool utility data.
53
INSPIRE Jamaica: Screening and Early Intervention System for Children and Families
Table 4.1
Part 2: CBRS Summary of Responses to Utility Statements*
Total Strongly
Agree
Agree No
Opinion
Disagree Strongly
Disagree
Missing
N N (%)
N (%)
N (%)
N (%)
N (%)
N (%)
In general,
questions clear
and easy to
understand
39 15
(38.5%)
23 (59%) 0 (0%) 1 (2.6%) 0 (0%) 0 (0%)
Questions
appropriate for
child and
family’s culture
39 11 (28 %) 27 (69%) 0 (0%) 1 (3%) 0 (0%) 0 (0%)
Completing tool
provided
meaningful
Information
39 10 (26%) 25 (64 %) 4 (10%) 0 (0%) 0 (0%) 0 (0%)
Would like to
use this tool
again
39 9 (23%) 29 (74%) 0 (0%) 1 (3%) 0 (0%) 0 (0%)
Question
appropriate for
child’s age
39 12
(30.8%)
26
(66.7%)
0 (0%) 1 (2.6%) 0 (0%) 1 (3%)
*Response to statements (e.g., In general, were the questions clear and easy to understand?) were
on a 5 point Likert Scale, (Strongly Agree, Agree, No Opinion, Disagree and Strongly Disagree).
Forty-nine utility surveys were collected on the School Readiness Skills. The following
table summarizes School Readiness Skills utility data.
54
INSPIRE Jamaica: Screening and Early Intervention System for Children and Families
Table 4.2
Part 3: School Readiness Domains Summary of Responses to Utility Statements*
Total Strongly
Agree
Agree No
Opinion
Disagree Strongly
Disagree
Missing
N N (%)
N (%)
N (%)
N (%)
N (%)
N (%)
In general,
questions clear
and easy to
understand
49 15 (31%)
33 (67%) 0 (0%) 1 (2%) 0 (%) 0 (%)
Questions
appropriate for
child and
family’s culture
49 12 (25%) 34 (69%) 0 (0%) 3 (6%) 0 (%) 0 (%)
Completing tool
provided
meaningful
Information
48 12 (25%) 31 (65%) 2 (4%) 2 (6%) 0 (%) 1 (2%)
Would like to
use this tool
again
48 10 (21%) 37 (77%) 1 (2%) 0 (0%) 0 (%) 1 (2%)
Question
appropriate for
child’s age
47 13 (28%) 33 (70 %) 0 (0%) 0 (0%) 1 (2%) 2 (4%)
*Response to statements (e.g., In general, were the questions clear and easy to understand?) were
on a 5 point Likert Scale, (Strongly Agree, Agree, No Opinion, Disagree and Strongly Disagree).
Additional written comments included the following responses to questions posed on the
utility survey.
Question 1: If you disagreed or strongly disagreed with any of the statements, please tell us
why.
Child Behavior Rating Scale
“I would change the scale and use: “Not all the time”, “Sometime”, “Most of the
time” as the scale was ambiguous and difficult to match the item and student
behavior.”
55
INSPIRE Jamaica: Screening and Early Intervention System for Children and Families
“Rubric wasn’t clear, a little confusing.”
School Readiness Skills
“I found this tool to be appropriate for the age group, but in the same breath
challenging to complete and manage a full class of over 30 students.”
“Not Age appropriate; some children just started school in September (have only
had one term of school). Just four this year.”
“The questions were in keeping with the basic achievement required for children
who are 5 years old. The instrument was quite comprehensive in covering the
affective and cognitive areas of development.”
Question 2: How would you change this tool to make it better?
Child Behavior Rating Scale
“I would not change this tool for now.”
“Some of the questions a bit vague. Didn’t give a very clear picture of children’s
needs.”
School Readiness Skills
“Change word number to numeral.”
“Make the questions a little more simpler/shorter.”
Question 3: We welcome further comments and suggestions. Feel free to write them below:
Child Behavior Rating Scale
“After using this tool make appropriate helpline for the children.”
“Based on the age and stage of development of 5yr olds the questions were
consistent with same.”
“I think this program is well needed and should be widely expanded.”
This will help me to know my children even better, not only the ones being
assessed.”
School Readiness Skills
“Felt was a very good assessment: it brings out what children could and could not
do so weaknesses could be worked on.”
“It is comparable to the assessment done at end of term to see how child is
progressing.”
56
INSPIRE Jamaica: Screening and Early Intervention System for Children and Families
“Questions opened eyes to child’s real and not assumed abilities.”
Summary of Utility Findings Utility data on the CBRS were quite positive. Almost all—97.5%
of respondents--strongly agreed/agreed that the questions on the CBRS are clear and easy to
understand. Ninety-seven percent of respondents strongly agreed/agreed that questions were
appropriate for their child and family’s culture. Ninety percent of respondents strongly
agreed/agreed that completing the tool provided meaningful information about the child; Ninety-
seven percent of respondents strongly agreed/agreed that they would like to use this screening
tool again. Current findings and anecdotal reports from the UWI research team suggest that the
CBRS will be an appropriate and useful screening tool for determining school readiness.
Utility data on the School Readiness Skills were also quite positive. Almost all--98% of
respondents--strongly agreed/agreed that the questions on the School Readiness Skills are clear
and easy to understand. Ninety-four percent of respondents strongly agreed/agreed that questions
were appropriate for their child and family’s culture. Ninety percent of respondents strongly
agreed/agreed that completing the tool provided meaningful information about the child. Ninety-
eight percent of respondents strongly agreed/agreed that they would like to use this screening
tool again. Current findings and anecdotal reports from the UWI research team suggest that the
School Readiness Skills will be an appropriate and useful screening tool for determining school
readiness.
Phase 2. Psychometric Data Collection on the School Readiness Screening,
Parts 2 and 3
The second phase of data collection involved collecting psychometric data on the field version of
the School Readiness Screening. Research staff from the University of West Indies assisted in
identifying 4-year preschool classrooms that provided a well-stratified a sample of Jamaican
families and children as possible, given financial and logistical constraints of the project. All
participating parents/caregivers of children in targeted settings were provided with information
about the School Readiness screening project and asked if they would be willing to participate.
As part of participation in the research, parents/caregivers received from their child’s teacher or a
UWI researcher a form asking for the child’s demographic information as well as a research
consent form. Teachers were also asked to complete a teacher demographic information form as
57
INSPIRE Jamaica: Screening and Early Intervention System for Children and Families
well as a research consent form. Part 2: Child Behavior Rating Scale (CBRS) and Part 3: School
Readiness Skills (SRS) of the School Readiness Screening were completed by teachers after a
researcher had provided them with an introduction to the screening project. Procedures ensuring
protection of human participants were approved by the University of West Indies and the
University of Oregon institutional review boards and were followed in all research phases.
The following data are based on 236 School Readiness screenings completed by teachers
of children between 42 and 66 months of age, in 4-year preschool classrooms.
Population Sample for the School Readiness Screening
Child and Family Information. The data analyses that are contained in this report are
based on 236 completed School Readiness screenings, which included both Part 1: CBRS and
Part 2: SRS. Each parent or caregiver who completed a questionnaire was asked to complete a
demographic form. Demographic data included information on the child’s age, gender, parish the
child resides, setting screening took place, who completed screening, and number of family
household resources (as a proxy of risks). It should be noted that there were 197 missing
demographics for Part 1: CBRS, and 195 missing demographics for Part 2: SRS; percentages
reported are based on the 41 CBRS data and 39 SRS data for which demographic information
was collected. The demographic data for the population sample are displayed in Tables 4.3 and
4.4. A copy of the demographic form can be found in the Appendix of this report.
Gender and age of child and parish where child resides. As shown in Table 4.3, the
gender distribution for the CBRS sample was 51% male and 49% female. Table 4.4 includes
information about the age of children included in the study. Because demographic information
was not available for these children, researchers looked at the age intervals of School Readiness
Screening Part 1:ASQ-J to determine approximate ages. These ages presume that children were
given the correct ASQ-J interval; birthdate information is not available to confirm this. Table 4.5
contains data on the parish where the child lived at the time of the data collection. The majority
of children lived in Kingston (51%), St. Andrew (26.8%), or St. Catherine (22%) parishes.
Settings Screenings Took Place. Table 4.6 contains data on the setting where the School
Readiness screening took place. A large percentage (93.7%) of School Readiness screenings
occurred in Early Childhood Educational and Primary school settings. 7.3% took place in “other”
settings. Approximately 68% of these settings were public and 32% were private settings. Close
to 98 percent (97.5%) of screenings were conducted in urban settings (15.4% of the total sample
58
INSPIRE Jamaica: Screening and Early Intervention System for Children and Families
was considered “inner” urban), while only 2.6% of screening occurring in rural settings. See the
Limitations of Study section for a discussion of the imbalance or urban/rural settings and
recommendations for future research on this tool.
Table 4.3
Gender of Children
CBRS School Readiness Skills
Frequency Valid Percent Frequency Valid Percent
Male 21 51.2 20 51.3
Female 20 48.8 19 48.7
Total 41 100.0 39 100.0
Missing 195 197
Total N 236 236
Note: Sample N’s were calculated separately for CBRS and School Readiness
Table 4.4
Age of Children by ASQ-J Interval
Frequency Valid Percent
45-51 18 8.0
51-57 109 48.0
57-66 100 44.0
Total 227 100.0
Missing 9
Total N 236
59
INSPIRE Jamaica: Screening and Early Intervention System for Children and Families
Table 4.5
Parish Where Child Resides
Table 4.6
Setting Where Screening Took Place
Household resources (child risk). Data on the number of household resources was
collected as a proxy of child risk status. Parents were asked to self-report household resources
from a list of 26 resources (e.g., telephone, car, water heater). A total number of resources was
entered for each family. These data are displayed in Table 4.6. A little less than twenty percent
(19.5%) reported having 5-8 resources, 34.2% reported having 9-12 resources; 29.3% reported
having 13-16 resources; 14.6% reported having 17-20 resources, and 2.4% reported having more
than 21 household resources.
CBRS School Readiness Skills
Frequency Valid Percent Frequency Valid Percent
Kingston 21 51.2 21 53.8
St Andrew 11 26.8 9 23.1
St Catherine 9 22.0 9 23.1
Total 41 100.0 39 100.0
Missing 195 197
Total N 236 236
CBRS School Readiness Skills
Setting Frequency Valid Percent Frequency Valid Percent
Early Childhood
Educational 35 85.4 35 89.7
Primary School 3 7.3 1 2.6
Other 3 7.3 3 7.7
Total 41 100.0 39 100.0
Missing 195 197
Total N 236 236
60
INSPIRE Jamaica: Screening and Early Intervention System for Children and Families
Teacher information. Demographic data were also collected from the preschool teachers
who completed School Readiness Screenings and included information on the teacher’s years of
experience working with preschool age children, setting where the screening took place, the
teachers’ job title or role, years of experience in her current job, age, level of education, degree
and additional college coursework. Data were collected from a total of 43 participating teachers,
between the ages of 27 and 61.
Table 4.7
Total Number of Household Resources
Years experience working with preschool children. Table 4.8 contains data on the number
of years of experience participating teachers had working with preschool Children. Only three
(7.3%) of teachers had 5 or fewer years of experience, while eight teachers (19.5%) had between
six and ten years; ten teachers (24.4%) had between 11 and 15 years; ten teachers had between
16 and 20 years; and only three teachers (7.3%) had over 30 years of experience working with
preschool children.
Setting where screening took place / Job title or role / Years in current job. Teachers
were also asked where they worked, their current job title or role, and the number of years they
had been in their current. Tables 4.9 through 4.11 present teacher responses. Twenty-five
# of
Resources Frequency Valid Percent
1-4 0 0.0
5-8 8 19.5
9-12 14 34.2
13-16 12 29.3
17-20 6 14.6
21-24 1 2.4
Total 41 100
Missing 195
Total N 236
61
INSPIRE Jamaica: Screening and Early Intervention System for Children and Families
(59.5%) of the teachers reported working in Public Early Childhood Educational Institutions,
while 17 (40.5%) reported working in Private Early Childhood Educational Institutions. The
majority of those participating were teachers (86%) with three Teacher Assistants (7%) and three
others, one principal, one Senior Teacher, and one Assistant Administrator/Supervisor. Eighteen
teachers (43.9%) reported being in their current job for 10 years or less, while 16 teachers (39%)
report working in their current job for between 11 and 20 years. Only seven teachers (17%)
indicated they had over 20 years of experience in their current job.
Table 4.8
Years of Experience Working with Preschool Children
# of Years Frequency Valid Percent
1-5 3 7.3
6-10 8 19.5
11-15 10 24.4
16-20 10 24.4
21-25 2 4.9
25-30 5 12.2
31-35 1 2.4
35-40 2 4.9
Total 41 100
Missing 2
Total N 43
62
INSPIRE Jamaica: Screening and Early Intervention System for Children and Families
Table 4.9
Where Do You Work
Table 4.10
Current Job Title or Role
Level of Education / Degree / College Coursework. Teachers also reported on their level
of education and coursework, illustrated in Tables 4.12 and 4.13 Thirteen of the teachers (31.7%)
had either an Associates or Bachelor’s Degree, while 18 others (43.9%) reported having some
college coursework. Twenty-five of the teachers (69.4%) reported that most of their college
coursework was related working with young children and families. It should be noted that
responses to this question were recorded for only 36 out of the 43 teachers.
Frequency Valid Percent
Public Early Childhood
Educational Institution 25 59.5
Private Early Childhood
Educational Institution 17 40.5
Total 42 100
Missing 1
Total N 43
Frequency Valid Percent
Teacher 37 86.0
Teacher Assistant 3 7.0
Other 3 7.0
Total N 43 100
63
INSPIRE Jamaica: Screening and Early Intervention System for Children and Families
Table 4.11
Years of Experience Working in Current Job
Table 4.12
Level of Education
# of Years Frequency Valid Percent
1-5 11 26.8
6-10 7 17.1
11-15 8 19.5
16-20 8 19.5
21-25 1 2.4
25-30 4 9.8
31-35 1 2.4
35-40 1 2.4
Total 41 100
Missing 2
Total N 43
Frequency Valid Percent
Primary School 1 2.4
Second School Diploma 9 22.0
Some college 18 43.9
Associate’s Degree (AA) 3 7.3
Bachelor’s Degree 10 24.4
Total 41 100
Missing 2
Total N 43
64
INSPIRE Jamaica: Screening and Early Intervention System for Children and Families
Table 4.13
College Coursework Related with Working with Young Children and their Families
Cutoff Scores and Percent Identified of Population for the School Readiness Screening
A main goal of Phase 2 was to establish cutoff scores for the School Readiness Screening
in order to help teachers to determine whether additional developmental evaluation is necessary,
and to assist in curriculum planning to support children’s readiness for primary school. As
discussed earlier, cutoff scores for Part 1: ASQ-J of the School Readiness Screening are
discussed in Chapter 3. Cutoff Scores for Part 2 and Part 3 are discussed below. In order to assist
with the development of cutoff scores, researchers intended to collect a known groups validity
sample of School Readiness Screenings on children who had been previously identified with
developmental delays by teachers and/or developmental pediatricians. However, due to limited
resources, the collection of this sample was not possible.
Cutoff Scores for Part 2: Child Behavior Rating Scale. The cutoff score for the CBRS
was set at 6 or more total concerns. This cutoff score was based on analyses of the means and
standard deviations for concern scores in the normative sample. The mean number of concerns
for children in the normative sample was 2.7, while the standard deviation was 3.15. The cutoff
score of 6 was set at one standard deviation above the mean. The percentage of children in the
normative sample identified for additional support or referral by this cutoff was 16.5%.
Cutoff Scores for Part 3: School Readiness Skills. An individual cutoff score for each of
the three areas of the School Readiness Skills (Approaches to Learning/Social Emotional, Early
Literacy and Early Numeracy) were established using the same method as the CBRS. As
illustrated in Table 4.14, cutoffs were set at one standard deviation below the mean for each area
Frequency Valid Percent
None 1 2.8
Some (25%) 6 16.7
Half (50%) 4 11.1
Most (75%) 25 69.4
Total 36 100
Missing 7
Total N 43
65
INSPIRE Jamaica: Screening and Early Intervention System for Children and Families
of the School Readiness Skills. Although one standard deviation below the mean in the Early
Numeracy Area is 8.4, this number was rounded up to a cutoff of 9 as the cutoffs are set as whole
numbers, with a score below the cutoff considered to be cause for follow-up. Using the
established cutoff scores 14.0% were below the cutoff score in the Approaches to
Learning/Social Emotional area, 15.3% were below in the Early Literacy area, and 15.7% were
below in the Early Numeracy area; 16.1% of children in the normative sample were below the
cutoff score in one or more areas.
Table 4.14
Means, Standard Deviations, Cutoff Scores and Percent Identified by Area of the School
Readiness Skills
Approaches to
Learning Early Literacy Early Numeracy
N Valid 236 236 236
Missing 0 0 0
Mean 13.44 13.97 12.24
Standard Deviation 3.88 5.01 3.84
Cutoff Score 10 9 9
Percent Identified 14.0 15.3 15.7
Reliability and Internal Consistency.
Researchers intended to examine test-retest reliability on both Part 1: CBRS and Part 2:
School Readiness Skills. However, limitations in terms of time and resources made it impossible
to collect test-retest data on both tools and the School Readiness Skills were prioritized given
that it was a newly-developed tool. In order to collect test-retest data, teachers were asked to
complete all three areas of the School Readiness Skills at Time 1 and again at Time 2, two to
three weeks later. A test-retest sample size of 50 was targeted; the actual sample size was 36. A
Pearson Product Correlation was used to analyze test-retest reliability, and results indicate strong
significant agreement in all three areas, as shown in Table 4.15.
Inter-observer reliability was also examined for the Early Literacy and Early Numeracy
areas of the School Readiness Skills. Both teachers and a researcher completed both areas
66
INSPIRE Jamaica: Screening and Early Intervention System for Children and Families
through observations and interactions with individual children during the same two-week period.
Researchers had hoped that the second rater could be someone familiar with the children and
their skills; however, each classroom had only one teacher and no assistants making it necessary
to utilize a researcher as a second observer. For this reason, the Approaches to Learning/Social
Emotional area was not included in the inter-observer study, as skills in this area require
observation over time by a rater with whom children have an ongoing relationship. For this
reason, it was assumed that inter-observer reliability would not be high in this area. The target
number of children for the inter-observer study sample was 50; this goal was met. Inter-observer
reliability was measured with Pearson Production Correlations, using total area scores. As
illustrated in Table 4.15, inter-observer correlations between total scores differed between the
two areas. The Early Literacy area had a correlation of .75, which is considered good, while the
correlation between total scores for the Early Numeracy area is .48, which is considered weak.
Both correlations were significant at the .01 level (2-tailed). Researchers also analyzed the
agreement between two observers, a teacher and a researcher, on classification, and found that
agreement was strong for both the Early Literacy (.90) and Early Numeracy areas (.90); 90% of
the time observers were in agreement as to whether a child was above or below the area cutoff
score.
Table 4.15
Summary of Test-Retest and Inter-Observer Reliability (Pearson Product Correlational Analyses)
Approaches to Learning/
Social Emotional Early Literacy Early Numeracy
Study N r N r N r
Test-retest 36 .85** 36 .75** 36 .70**
Inter-
observer 50 .75** 50 .48**
**Correlation is significant at the .01 level (2-tailed).
Internal consistency also was examined for all three areas of the School Readiness Skills
(Approaches to Learning/Social Emotional, Early Literacy and Early Numeracy) as well as for
the CBRS by examining the relation between item scores using correlational analysis and
67
INSPIRE Jamaica: Screening and Early Intervention System for Children and Families
Cronbach’s coefficient alpha. The standardized alpha for the Approaches to Learning (.81), Early
Literacy (.89), and Early Numeracy (.87) areas indicated strong internal consistency for all three
areas. Internal consistency was also examined for the CBRS, and the standardized alpha was .86,
also indicating strong internal consistency.
Limitations and Future Directions.
There are several limitations to this study that should be considered when implementing
the School Readiness Screening and when considering future research directions. It is important
to note that this was an ambitious study, conducted within an exceptionally short time frame and
with limited resources. As discussed earlier, demographic information was not collected from
197 families, so percentages reported are based on the 41 families from whom demographic
information was collected. Data were also missing for some individual questions on gathered
demographic forms. Because demographic data are missing, it is not clear whether the normative
sample is representative of the general Jamaican population. Collected demographics indicate
that the majority of school readiness screening data was collected in three parishes: Kingston
(51.2%), Saint Andrew (26.8%), and Saint Catherine (22%). As also discussed earlier, 97.5% of
the School Readiness screenings for which demographic information was collected took place in
urban settings, while only 2.6 % took place in rural settings. Cutoff scores were derived based on
means and standard deviations and presumed a normative sample that was stratified in terms of
number of risk factors, and included households with a low, medium and high number of risk
factors. However, since demographic information was limited, and researchers did not have the
ages of participating children, researchers were unable to examine the relationship between mean
scores and age. Without that demographic information it was also not possible to make
conclusions about the degree to which the sample represents the general Jamaican population.
Future studies on the School Readiness Screening should include a representative sample
of the Jamaican population. As also discussed above, future research should also investigate the
validity of established cutoff scores and ensure that the tool is measuring similar skills as other
established school readiness tools as well as identifying children most in need of support.
68
Project Title: INSPIRE Jamaica Child ID______________
University of Oregon IRB # 11162012.020 Research Assistant ID______________
Utility Survey
Please continue on the next page
1. Who is providing this feedback?
☐ Parent (1) ☐ Health Care Provider (2)
☐ Teacher (3) (ID#__________) ☐ Teacher’s Assistant (4) (ID#____________)
☐ Other: (5) ______________________
2. Which tool is feedback focused on? (Tick one response):
☐ SWYC (1) ☐ ASQ (2) ☐ Family Support (3) ☐ Teacher portion School Readiness (4)
2a. If ASQ, indicate interval completed (e.g., 12 months ______________ )
2b. If SWYC, indicate interval completed (e.g., 12-14 months ___________)
3. How long did it take to complete the tool (e.g., 10 minutes _________)?
Please provide your opinion about the following statements:
4. “In general, questions were clear and easy to understand.” (Tick one response)
☐Strongly agree (4) ☐Agree (3) ☐No opinion (2) ☐Disagree (1) ☐Strongly disagree (0)
5. “In general, questions were appropriate for child and/or family’s culture.” (Tick one)
☐Strongly agree (4) ☐Agree (3) ☐No opinion (2) ☐Disagree (1) ☐Strongly disagree (0)
6. “Completing the tool gave me meaningful information and child and/or family’s current
abilities and needs.” (Tick one response)
☐Strongly agree (4) ☐Agree (3) ☐No opinion (2) ☐Disagree (1) ☐Strongly disagree (0)
7. “ I plan (or would like) to use this tool again.” (Tick one response)
☐Strongly agree (4) ☐Agree (3) ☐No opinion (2) ☐Disagree (1) ☐Strongly disagree (0)
69
Thank you for participating in this study!
22 January 2013
If providing feedback on Family Support tool SKIP to question #9
8. “In general, questions were appropriate for child’s age” (Tick one response)
☐Strongly agree (4) ☐Agree (3) ☐No opinion (2) ☐Disagree (1) ☐Strongly disagree (0)
9. If you disagreed or strongly disagreed with any of the statements, please tell us why.
10. How would you change the tool to make it better?
11. We welcome further comments and suggestions. Feel free to write them below:
For Teachers/Professional providers (not intended for parents to complete)
12. How many times have you completed the above tool? _______ _(number of times completed)
13. Do you have a preferred way of completing the tool? For example, do you have an opinion
about where, how, or when it is administered?
☐No (0) ☐Yes (1) (if yes, please tell us more) ________________________
_________________________________________________________________________
_________________________________________________________________________
****************************************************************************************
70
Project Title: INSPIRE Jamaica Child ID______________
University of Oregon IRB # 11162012.020 Research Assistant ID_____________
Child and Family Information: Parent Section Date Screening (s) Completed: ______________________________________ 1. What tool(s) were completed? (Tick all that are appropriate):
☐ Family Support Screening tool ☐ SWYC ☐ ASQ ☐ Teacher portion School Readiness (enter teacher ID #________________)
2. Child’s Date of Birth: _____________________
2a. For children under 2 years, was child 3 or more weeks premature? ☐ No (0) ☐ Don’t Know (2) ☐ Yes (1) (If yes, weeks premature?___________)
3. Child’s Gender: ☐ Male (1) ☐ Female (2)
4. Parish where child lives: (Tick where appropriate):
☐ Kingston (001) ☐ St Andrew (002) ☐ St Thomas (003) ☐ Portland (004) ☐ St Mary (005) ☐ St Ann (006) ☐ Trelawny (007) ☐ St James (008) ☐ Hanover (009) ☐ Westmoreland (010) ☐ St Elizabeth (011) ☐ Manchester (012) ☐ Clarendon (013) ☐ St Catherine (014) ☐ Don’t Know (99)
5. Setting where screening took place? (Tick where appropriate):
☐ Antenatal (1) ☐ Early Childhood Educational Institution (6) ☐ Clinic (2) ☐ Primary School (7) ☐ PATH Home Visit (3) ☐ Nursery (8) ☐ WCC (4) ☐ Other (9) (please specify__________)
☐ Pediatrician (5)
5a. Was the setting public or private? (Tick where appropriate): ☐ Public (government) (1) ☐ Private (2) ☐ Don’t Know (3) 5b. Setting geographic location (Tick where appropriate):
☐ Rural (1) ☐ Urban (2) ☐ Inner City Urban (3) ☐ Don’t Know (4)
6. Who completed tools (answered questions)? (Tick where appropriate): ☐ Mother (1) ☐ Aunt (6) ☐ Brother (11) ☐ Father (2) ☐ Uncle (7) ☐ Sister (12) ☐ Grandmother (3) ☐ Stepmother (8) ☐ Other-family (13) (specify_______________) ☐ Grandfather (4) ☐ Stepfather (9) ☐ Non-family (14) (specify________________) ☐ Teacher (5) ☐ Teaching Assistant (10)
71
Project Title: INSPIRE Jamaica Child ID______________
University of Oregon IRB # 11162012.020 Research Assistant ID_____________
7. Who supported the completion of the questionnaires? (Tick where appropriate):
☐ No one (1) ☐ Research Assistant (3) ☐ Health Care Provider (5) ☐ Teacher (2) ☐ Teaching Assistant (4) ☐ Other (6) (specify_____________)
8. Which of the following resources does the family have in their household? (Tick all that apply):
☐ Television ☐ Cars or other vehicles ☐ Fans ☐ Water heater (solar) ☐ Cable/Satellite connection ☐ Telephone ☐ Gas Stove ☐ Generator ☐ Refrigerator ☐ VCR/DVD player ☐ Electric Stove ☐ Video equipment ☐ Freezer ☐ Computer ☐ Air Conditioners ☐ Game boy/Play station
☐ Living Room Set ☐ Internet connection ☐ CD/DVD Burner ☐Other electrical equipment ☐ Stereo Equipment ☐ Radio ☐ Water tank (toaster/microwave/blender) ☐ Washing Machine ☐ Sewing Machine ☐ Water heater (electric)
Enter total number of ticked items______
Section II. Additional Child Information
1. Date Screening (s) Completed: __________________________
2. Diagnosis: Does your child have a known delay or disability, or a medical condition with a high probability of
developmental delay (e.g., Down syndrome)?
☐ No (0) ☐ Don’t Know (2)
☐ Yes (1) If yes, what is disability/diagnosis?_______________________________________
_____________________________________________________________________________
72
INSPIRE Jamaica: Screening and Early Intervention System for Children and Families
Appendix C
Technical Report References
Abidin, R. R. (1983). Parenting Stress Index (PSI), 4th
Edition. Charlottesville, VA: Pediatric
Psychology Press.
American Academy of Pediatrics. (2006). Identifying infants and young children with developmental
disorders in the medical home: An algorithm for developmental surveillance and screening.
Pediatrics, 118(1), 405-420.
Ball, J., & Janyst, P. (2008). Enacting research ethics in partnerships with indigenous communities in
Canada: "Do it in a good way." Journal of Empirical Research on Human Research Ethics,
3(2), 33-52
Bian, X., Yao, G., Squires, J. Wei, M., Chen, C., & Fang, B. (2010). Studies of the norm and
psychometric properties of Ages and Stages Questionnaires in Shanghai children. Zhonghua Er
Ke Za Shi. Chinese Journal of Pediatrics, 48(7), 492–496.
Bronson, M., Goodson, B., Layzer, J., & Love, J. (1990). Child Behavior Rating Scale. Cambridge,
MA: ABT Associates.
Cronbach, L. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 279-334.
Dionne, C., Squires, J., & Leclerc, D. (2004). Psychometric properties of a developmental screening
test. Using the Ages and Stages Questionnaires (ASQ) in Quebec and the U.S. Journal of
Intellectual Disability Research, 48(4-5), 408.
Heo, K., Squires, J., & Yovanoff, P. (2008). Cross-cultural adaptation of a preschool screening
instrument: Comparison of Korean and U.S. populations. Journal of Intellectual Disability
Research, 52(3), 195-206.
Janson, H., & Squires, J. (2004). Parent-completed developmental screening in a Norwegian
population sample: A comparison with U.S. normative data. Acta Paediatrica, 93(11), 1525-
1529.
Kapci, E. G., Kucuker, S., & Uslu, R. I. (2010). How applicable are Ages and Stages Questionnaires
for use with Turkish children? Topics in Early Childhood Special Education, 30, 176-188.
Musquash, C., & Bova, D. (2007). Cross cultural assessment and measurement issues. Journal of
Developmental Disabilities, 13(1), 53-66.
Radloff, L. S. (1977). The CES-D scale: a self-report depression scale for research in the general
population. Applied Psychological Measurement, 1:385-401.
73
Sektnan, M., McClelland, M. M., Acock, A., & Morrison, F. J. (2010). Relations between early family
risk, children’s behavioral regulations, and academic achievement. Early Childhood Research
Quarterly, 25(4), 464-479.
Squires, J., Bricker, D. & Twombly, E. (2002). Ages and stages questionnaires: Social-Emotional:
A parent-completed child-monitoring system. Baltimore: Paul Brookes.
Squires, J., Bricker, D., Twombly, E., & Potter, L. (2009). Ages and stages questionnaires user’s guide
3rd
edition: A parent-completed child-monitoring system. Baltimore, MD: Paul Brookes
Publishing.
74