eqao alignment analysis - university of michiganmayess/sarah_mayes:_teaching...the eqao primary and...
TRANSCRIPT
ANALYSIS OF EQAO MATHEMATICS ASSESSMENTS
Mayes_03.29 EDUC783-Winter12
1
The EQAO Primary and Junior Mathematics Assessments: An Alignment and Content Analysis
Sarah Mayes
University of Michigan
ANALYSIS OF EQAO MATHEMATICS ASSESSMENTS
Mayes_03.29 EDUC783-Winter12
2
The EQAO Primary and Junior Mathematics Assessments: An Alignment and Content Analysis
As the results of large-scale assessments gain increased attention at local, regional,
national, and international levels, it is becoming even more important that they be of the highest
quality possible (La Marca, 2001). The multiple roles that assessments play dictate factors that
must be considering when analyzing their effectiveness. One of these roles is to define curricula:
teachers, students, and other interested parties look to large-scale assessments as an indicator of
the content and cognitive skills that students are supposed to learn (Herman, 2004).
Curricula are reflected in many forms across an educational system, including official
regional standards, textbooks, the material that is taught, and assessments. Alignment describes
the relationship between two or more of these curricula and measures the extent to which they
agree. Researchers have studied the alignment between assessments and textbooks (Chandler &
Bronsan, 1995), between assessments and instruction (Porter, 2002), and between assessments
and standards (Roach, Elliott, & Webb, 2005; Webb, 1999; Wixson, Fisk, Dutro, & McDaniel,
2002) across various subjects.
Comparing standards and assessments has become increasingly important in recent years
as localities adopt various standards-based curricula and large-scale assessments purporting to
measure these curricula (Herman, Webb, & Zuniga, 2007). Achieving good alignment between
standards and assessments is not only essential for accountability (La Marca, 2001), validity of
test scores (Rothman, 2003), and providing information to the public (Herman, Webb, & Zuniga,
2007), but also for creating a coherent education system and sending consistent messages to
students and to teachers. If there is not good alignment, teachers may "teach to the test" and
cover only what is assessed (Herman, 2004). Further, some researchers believe that better
aligning tests with standards can increase student learning (Farenga, Joyce, & Ness, 2002) and
ANALYSIS OF EQAO MATHEMATICS ASSESSMENTS
Mayes_03.29 EDUC783-Winter12
3
minimize the effects of factors such as socio-economic status, parental education, and home
support on students' performance (Mohamud & Fleck, 2010).
Assessments should not only be aligned with the standards they are supposed to measure,
but, like other aspects of the curriculum, they should also follow the practices of the academic
subjects which they assess. Precision and clarity, for instance, are vital parts of mathematical
practice communication; thus, tests evaluating mathematics should have questions that are
correct and instructions that are clear.
This paper reports the results of an analysis of ten recent elementary-level large-scale
assessments in Ontario administered by the Education Quality and Accountability Office
(EQAO). The research questions guiding this analysis were:
1. How well aligned to The Ontario Curriculum (Ontario Ministry of Education,
2005) are the EQAO Primary and Junior mathematics assessments?
2. Are the items on EQAO Primary and Junior mathematics assessments
mathematically correct?
As discussed above, the alignment and mathematical integrity of large-scale assessments is
important. The answers to these questions can provide guidelines for improving the EQAO
assessments while clarifying the current role of the assessments in Ontario’s education system.
The following section provides further background on The Ontario Curriculum (Ontario
Ministry of Education, 2005) and on EQAO necessary to understand the analysis conducted.
The Methods section outlines the details about how the analyses were conducted while the
Results section answers the research questions. Finally, the Discussion section lays out some
implications of this analysis and suggests how the quality of EQAO assessments might be
improved.
ANALYSIS OF EQAO MATHEMATICS ASSESSMENTS
Mayes_03.29 EDUC783-Winter12
4
Background
The first part of this section provides background information about the provincial
mathematics curriculum in Ontario while the second part outlines the administration, purposes,
and structure of the EQAO assessments.
The structure of the Ontario mathematics curriculum
Students in Ontario generally attend elementary school from grades 1 through 8 and high
school from grades 9 through 12. The current version of the provincial mathematics curriculum
for elementary schools, The Ontario Curriculum, Grades 1-8: Mathematics (Ontario Ministry of
Education, 2005) was implemented in September 2005.1
There are two types of expectations in the Ontario mathematics curriculum. The Process
Expectations are the same for every grade level and include: problem solving; reasoning and
proving; reflecting; selecting tools and computational strategies; connecting; representing; and
communicating. The Specific (content) Expectations are divided into the following five strands:
Number Sense and Numeration; Measurement; Geometry and Spatial Sense; Patterning and
Algebra; and Data Management and Probability. Within each strand, expectations are further
classified into two to three sub-strands (see Table 1). The process expectations are to be
incorporated into each of the content expectations at a grade-appropriate level.
1 Ontario curriculum documents are available on the Ministry of Education’s website, http://www.edu.gov.on.ca.
ANALYSIS OF EQAO MATHEMATICS ASSESSMENTS
Mayes_03.29 EDUC783-Winter12
5
Table 1
Strands and sub-strands in the of The Ontario Curriculum for grades 3 and 6
Strand Grade 3 Sub-strands Grade 6 Sub-strands NSN: Number Sense and
Numeration NSN1: Quantity
Relationships NSN2: Counting NSN3: Operational Sense
NSN1: Quantity Relationships
NSN3: Operational Sense NSN4: Proportional
Relationships M: Measurement M1: Attributes, Units, and
Measurement Sense M2: Measurement
Relationships
M1: Attributes, Units, and Measurement Sense
M2: Measurement Relationships
GSS: Geometry and Spatial Sense
GSS1: Geometric Properties GSS2: Geometric
Relationships GSS3: Location and
Movement
GSS1: Geometric Properties GSS2: Geometric
Relationships GSS3: Location and
Movement PA: Patterning and
Algebra PA1: Patterns and
Relationships PA2: Expressions and
Equality
PA1: Patterns and Relationships
PA2: Variables, Expressions, and Equations
DMP: Data Management and Probability
DMP1: Collection and Organization of Data
DMP2: Data Relationships DMP3: Probability
DMP1: Collection and Organization of Data
DMP2: Data Relationships DMP3: Probability
Although most of the Ontario curriculum document outlines specific expectations, topics
such as classroom assessment, accommodation of exceptional students, technology, and
integration of other subjects are also discussed. Neither this document nor other publications by
the Ministry of Education give any explicit guidance on the relative importance of strands, sub-
strands, or expectations. This implies that all strands, sub-strands, and expectations are equally
important; this assumption will be made for the alignment analysis described here.
Unless otherwise specified, in this document, "the curriculum" will refer to The Ontario
Curriculum, Grades 1-8: Mathematics (Ontario Ministry of Education, 2005).
ANALYSIS OF EQAO MATHEMATICS ASSESSMENTS
Mayes_03.29 EDUC783-Winter12
6
The EQAO Assessments
The Education Quality and Accountability Office (EQAO), established in 1996, is a
Crown agency of the government of Ontario. EQAO's mandate is to conduct province-wide
assessments of performance in mathematics, reading, writing, and literacy. There are four
separate mathematics assessments conducted each year: the Primary Assessment of
Mathematics written at the end of grade 3; the Junior Assessment of Mathematics written at the
end of grade 6; and two Grade 9 Assessments of Mathematics taken at the completion of a grade
9 math course.2 These assessments test the math skills that students are expected to have gained
up to that point, as specified in the curriculum.
The Primary and Junior Assessments of Mathematics currently consist of 7 short-answer
and 29 multiple-choice questions each.3 Students writing the Primary assessment may not use a
calculator or manipulatives for the first six multiple-choice questions; apart from this, calculator
use is permitted throughout the other mathematics assessments.4
The EQAO assessments are criterion-referenced and use the same four-level rubric that is
used for all subjects and grade levels in Ontario (for example, level 3 denotes performance at the
provincial standard). The analysis of the results of the EQAO tests uses the Item Response
Theory model known as the 2 Parameter Logistic model, which takes into account both item
difficulty and the relationship between student scores on the test and responses to the item
(Kozlow, 2007).
Every student who writes an EQAO assessment is given an Individual Student Report
that states the overall level that they achieved on the assessment. Detailed provincial, school
2 Copies of the EQAO assessments are available on the agency’s website, http://www.eqao.com. 3 Different test structures have been used in the past but all assessments considered in this analysis are of this current form. 4 Schools are required to ensure that students have calculators during the tests.
ANALYSIS OF EQAO MATHEMATICS ASSESSMENTS
Mayes_03.29 EDUC783-Winter12
7
board, and school reports are available on EQAO’s website. These reports include the number of
students within the province, school board, or school achieving each level on the assessment
along with a summary of the results of the student questionnaire.
According to the EQAO (2012), the results of these tests are used by the provincial
government to "make local schools and school boards accountable" and by administrators at all
levels "for effective system-improvement planning." Although the tests are not as high-stakes as
some similar American assessments and are in no way tied to funding, student placement, or
teacher evaluation, there is public pressure for schools and school boards to perform well.5
Method
This section will be organized as follows. First, the Sampling sub-section describes
which assessments were chosen to be part of the analyses and why. The Analysis subsections
detail the procedure for matching assessment items to expectations in the curriculum, the
textbook coding procedure, and the alignment criteria used. Finally, the Quality of Data sub-
section justifies two aspects of the assessment and textbook coding procedures.
Sampling
Although EQAO tests have been administered each year since 1998, the curriculum and
the structure of the tests has changed since that time. A revised Ontario Mathematics Curriculum
document was implemented in September 2005 and the Primary Assessment did not take its
current form in regards to the use of calculators and manipulatives until 2007. To ensure that all
tests evaluated had a similar structure, I restricted the analysis to the five Primary and the five
Junior mathematics assessments administered from 2007 through 2011.
I chose not to analyze the grade 9 mathematics assessments because the high school
5 The Fraser Institute, for example, uses the results of the EQAO assessments to rank high schools in Ontario.
ANALYSIS OF EQAO MATHEMATICS ASSESSMENTS
Mayes_03.29 EDUC783-Winter12
8
mathematics curriculum is significantly different from the elementary curriculum. For example,
the strands and sub-strands of high school courses are course-specific and there are two different
grade 9 mathematics courses that students can take. As a result, the structure of the EQAO grade
9 mathematics assessments is not the same as the structure of the elementary assessments.
Analysis: Item Coding Process
I matched specific expectations from the grade 3 (respectively, grade 6) mathematics
curriculum to each item (or problem) from the Primary (respectively Junior) mathematics
assessments. More than one curriculum expectation was matched to some items if skills from
multiple expectations were needed to solve the problem or if significantly different strategies
could be used to solve the problem.6 Each match of an expectation to an assessment item is
called a hit for that expectation.
The following assumptions were made in the coding process.
• Students are likely to use the easiest or most direct method available that they know. For
example, students use calculators, when available, rather than doing a computation by
hand.
• Different students may use knowledge from different expectations to solve a given
problem. Consider, for example, question 30 from the 2007 Primary Assessment:
Steven earns $5 for every bundle of newspapers he delivers. He wants to buy a
game that costs $18. How many bundles of newspapers does Steven need to
deliver to earn enough money to buy this game?
A grade 3 student could use either skip counting or multiplication to solve this problem.
In the item matching, expectations corresponding to both methods would be matched to
6 The decision to match some items to multiple expectations is consistent with the alignment model developed by Webb (2005) that I use in this analysis.
ANALYSIS OF EQAO MATHEMATICS ASSESSMENTS
Mayes_03.29 EDUC783-Winter12
9
this item even though any one student would only need to use one strategy to solve the
problem.
Assessment items were matched to expectations from grades 3 and 6 rather than to
expectations from all primary or junior grades; this choice was made for two reasons. First,
nearly all items on the Primary (Junior) Assessments were at the grade 3 (grade 6) level rather
than a lower grade level. Second, among those items that could have been matched to an
expectation at a lower grade level, there was almost always a grade 3 (grade 6) expectation that
included the expectation from a lower grade either explicitly or as a prerequisite.
The exception to this rule came from a set of expectations in the Location and Movement
sub-strand of the Junior curriculum. The following expectation appears in each junior grade with
the blanks filled in with translations grade 4, reflections in grade 5, and rotations in grade 6:
"identify, perform, and describe ______ using a variety of tools" (Ontario Ministry of Education,
2005). I decided to classify items requiring students to identify, perform, and describe
translations and reflections into the similar grade 6 expectation that included only rotations.
During the matching process it became clear that it was necessary to add one expectation
to the NSN3: Operations Sense sub-strand in the grade 3 curricula. This additional expectation
states "relate addition and subtraction to real-life situations.” This expectation did not appear in
the curriculum for any of the primary grades and is analogous to the grade 3 expectation: "relate
multiplication of one-digit numbers and division by one-digit divisors to real-life situations,
using a variety of tools and strategies" (Ontario Ministry of Education, 2005). While one might
argue that the ability to “relate addition and subtraction to real-life expectations” should be
considered part of the problem-solving process expectation, I felt that it was necessary to include
this expectation as a content standard due to the pervasiveness of this skill in the primary-level
ANALYSIS OF EQAO MATHEMATICS ASSESSMENTS
Mayes_03.29 EDUC783-Winter12
10
assessments studied and to make the treatment of addition and subtraction consistent with that of
multiplication and division.
During the matching process, I flagged test items containing unclear wording, vague
instructions, or incorrect statements.
Analysis: Textbook Coding Process
Ontario's Trillium List includes all of the textbooks approved for use in the province's
classrooms.7 It includes three grade 3 mathematics texts and two grade 6 mathematics texts.
Textbook sections in one approved grade 3 text (Nelson Mathematics 3) and both approved grade
6 texts (Mathematics Makes Sense 6 and Nelson Mathematics 6) were counted and categorized as
focusing on one of the five content strands from the Ontario curriculum.8 Sections with a focus
in more than one strand were recorded as fractions to the nearest quarter of a section. Review
sections and sections focused on process standards without a specific content were excluded.
Analysis: Alignment Criteria Used
Each of the ten assessments was judged for alignment to the Ontario curriculum using the
following three criteria from the model developed Webb (2005): categorical concurrence, range-
of-knowledge, and balance of representation.
Categorical Concurrence
One way to judge the alignment of an assessment to a curriculum is to measure whether
the assessment measures all of the content categories that make up the curriculum. For example,
a test containing only questions on arithmetic is not aligned to the Geometry and Spatial Sense
strand. Webb says that the categorical concurrence criterion is met for a content category if the
assessment adequately measures it. The current analysis measures categorical concurrence at
7 The Trillium List can be viewed online at http://www.edu.gov.on.ca/trilliumlist/ 8 A ‘section’ refers to a section within a chapter as defined by the textbook authors.
ANALYSIS OF EQAO MATHEMATICS ASSESSMENTS
Mayes_03.29 EDUC783-Winter12
11
both the strand and the sub-strand levels and assumes that at least there needs to be at least six
hits for a category in order for the assessment to have acceptable categorical concurrence with
that category.9 That is, six hits are needed to adequately assess whether a student has mastered
the given skill (Webb, 2005). Note that this criterion is more difficult to meet on the sub-strand
level than on the strand level.
Range-of-Knowledge
Assessments that are aligned with curricula should measure the same span of knowledge
as the curricula itself. For example, suppose that the curriculum has ten expectations in the
Measurement Relationships sub-strand and that only one of these expectations relates to finding
areas. If every item on an assessment drawing from the Measurement Relationships sub-strand
relates to finding areas, this assessment is not aligned with the curricula for the sub-strand
because it does not cover the same breadth as the curricula. The range-of-knowledge criterion is
met for a strand or a sub-strand on an assessment if the span of knowledge required to complete
the assessment is similar to the span of knowledge expected by the curricula. For this standard to
be met for a strand or sub-strand on an assessment, at least 50% of the expectations within a
strand or a sub-strand need to be hit by assessment questions.
Balance of Representation
A well-aligned assessment should test each of the expectations which are hit an equal
number of times. The balance of representation criterion measures how equally assessment
content is distributed among the standards that are hit. The balance index for an assessment with
respect to a given strand or sub-strand is given by
9 Recall that each match of an assessment item to a curriculum expectation is a ‘hit’ for that expectation.
ANALYSIS OF EQAO MATHEMATICS ASSESSMENTS
Mayes_03.29 EDUC783-Winter12
12
where O is the total number of objectives hit within the strand or sub-strand, is the number of
hits corresponding to an expectation k, and H is the total number of hits within the strand or the
sub-strand. An index value of 1 indicates perfect balance and that hits are equally distributed
among the hit expectations while small index values indicate poor balance. Webb says that the
assessment satisfies the balance of representation criterion for a given standard or substandard if
the balance index is greater than or equal to 0.7.
The range-of-knowledge criterion and balance of representation criterion are
complementary and equally necessary to determine how well test items are distributed among
curriculum expectations. For example, consider a curriculum strand containing 20 expectations.
On one assessment, each of 8 expectations is hit exactly three times and no other expectations
are hit. In this case the range-of-knowledge correspondence is 40% (8/20) while the balance
index is 1. Now consider a second assessment where 15 of the 20 expectations are hit. One of
the expectations is hit 10 times while the remaining 14 expectations are hit only once. In this
case the range-of-knowledge is 75% (15/20) while the balance index is 0.7. Therefore, it is
possible to have assessments with very weak range-of-knowledge but perfect balance of
representation, as in the first case, and also assessments with strong range-of-knowledge and
weaker balance of representation, as in the second case.
Depth of Knowledge
Webb also includes a fourth alignment criterion, depth-of-knowledge, that compares the
complexity of knowledge stated in an expectation to the complexity of knowledge required to
complete a question on the assessment. To measure this criterion each content expectation must
ANALYSIS OF EQAO MATHEMATICS ASSESSMENTS
Mayes_03.29 EDUC783-Winter12
13
be assigned a level according to whether it includes recall, skills/concepts, strategic thinking, or
extended thinking. However, the Ontario curriculum includes process expectations similar to
Webb’s depth-of-knowledge levels that are to be integrated into every content expectation and so
it is impossible to assign complexity of knowledge scores to the content expectations in the
curriculum. Therefore, this criterion was not used in the current analysis of EQAO tests.
Quality of the Data
Although I was the only reviewer matching expectations to test items, an Ontario teacher
completed the matching procedure for one Primary Assessment and one Junior Assessment.
This reviewer’s matches agreed with mine for all but one question; this indicates that the data
used for the alignment analysis was reasonably accurate. Further, this reviewer looked at all
items that I flagged for problematic mathematical content and agreed with my concerns on each
of the items.
In the textbook analysis, counting sections provided almost the same information as
counting by pages, because each of the textbooks had a standard section length (nearly all
sections in Nelson Mathematics were two pages while in Mathematics Makes Sense nearly all
sections were three pages). I chose to count sections because it is likely that teachers planning
lessons around a textbook will follow the sections in the textbook.
Results
This section presents the results of the analyses conducted on the Primary and Junior
EQAO tests. The first sub-section details the results of the alignment analysis, the second sub-
section details the results of the textbook analysis, and the third sub-section discusses the results
of the mathematical integrity analysis.
ANALYSIS OF EQAO MATHEMATICS ASSESSMENTS
Mayes_03.29 EDUC783-Winter12
14
Alignment Analysis
Tables 2 and 3 summarize the results of the alignment analyses; detailed results are given in
Appendix. For each strand and sub-strand in the curriculum, these tables show the percentage of
the five Primary and the five Junior assessments satisfying each alignment criterion. The
following sub-sections, organized by alignment criterion, elaborate on the information found in
these tables.
ANALYSIS OF EQAO MATHEMATICS ASSESSMENTS
Mayes_03.29 EDUC783-Winter12
15
Table 2
Summary of results of alignment analyses of EQAO Primary Assessments 2007-2011
Strands and Sub-Strands % of Assessments with
acceptable Categorical Concurrence
% of Assessments with acceptable Range of
Knowledge
% of Assessments with acceptable Balance of
Representation NSN: Number Sense and Numeration 100 80 100
NSN1: Quantity Relationships 0 0 100 NSN2: Counting 0 100 100
NSN3: Operational Sense 100 100 100 M: Measurement 100 20 100
M1: Attributes, Units, and Meas. Sense 60 40 100 M2: Measurement Relationships 0 60 100
GSS: Geometry and Spatial Sense 60 20 100 GSS1: Geometric Properties 0 20 100
GSS2: Geometric Relationships 0 0 100 GSS3: Location and Movement 0 80 100
PA: Patterning and Algebra 100 80 100 PA1: Patterns and Relationships 40 80 100
PA2: Variables, Expressions, and Equations 0 60 100 DMP: Data Management and Probability 40 60 100
DMP1: Collection and Organization of Data 0 20 100 DMP2: Data Relationships 0 40 100
DMP3: Probability 0 100 100
Note: N=5
ANALYSIS OF EQAO MATHEMATICS ASSESSMENTS
Mayes_03.29 EDUC783-Winter12
16
Table 3
Summary of results of alignment analyses of EQAO Junior Assessments 2007-2011
Strands and Sub-Strands % of Assessments with acceptable Categorical
Concurrence
% of Assessments with acceptable Range of
Knowledge
% of Assessments with acceptable Balance of
Representation NSN: Number Sense and Numeration 100 0 100
NSN1: Quantity Relationships 0 20 100 NSN3: Operational Sense 0 0 100
NSN4: Proportional Relationships 0 100 100 M: Measurement 100 100 100
M1: Attributes, Units, and Meas. Sense 0 80 100 M2: Measurement Relationships 100 100 100
GSS: Geometry and Spatial Sense 100 100 100 GSS1: Geometric Properties 0 100 100
GSS2: Geometric Relationships 0 80 100 GSS3: Location and Movement 0 100 100
PA: Patterning and Algebra 100 100 100 PA1: Patterns and Relationships 40 80 100
PA2: Variables, Expressions, and Equations 0 60 100 DMP: Data Management and Probability 100 60 100
DMP1: Collection and Organization of Data 0 40 100 DMP2: Data Relationships 0 20 100
DMP3: Probability 0 60 100
Note: N=5
ANALYSIS OF EQAO MATHEMATICS ASSESSMENTS
Mayes_04.20 EDUC783-Winter12
25
Total Hits
Because multiple expectations could correspond to a single test item, the total
number of hits was greater than 36, the number of assessment items. The total number of
hits ranged from 43 to 49 on Primary assessments and from 39 to 42 on Junior
assessments (see Table 4). The greater number of hits for the primary-level assessments
is due to two factors. First, less knowledge is assumed at the grade 3 level than at the
grade 6 level and expectations are more specific at the lower level. Second, six questions
on each of the Primary assessments are to be done without access to calculators so that an
additional expectation related to computation are often matched to these questions;
calculator use is permitted throughout the Junior assessment.
Table 4
Total number of hits on from the analysis of each assessment
Assessment Year 2007 2008 2009 2010 2011 Total # of Hits on Primary Assess. 46 44 43 43 49 Total # of Hits on Junior Assess. 40 42 40 39 39
Categorical Concurrence
Only the 2011 Primary Assessment achieved an acceptable level of categorical
concurrence for all of the strands; each of the other Primary assessments had at least one
strand with six or fewer hits (see Tables A1.1 to A1.5 in Appendix 1). All of the Primary
assessments had acceptable categorical concurrence levels for the Number Sense and
Numeration, Measurement, and Patterning and Algebra strands while only 3 of the 5 of
the tests had satisfactory levels for the Geometry and Spatial Sense strand and only 2 had
satisfactory levels for the Data Management and Probability Strand. Only three of the
sub-strands had acceptable categorical concurrence on any of the Primary assessments:
ANALYSIS OF EQAO MATHEMATICS ASSESSMENTS
Mayes_04.20 EDUC783-Winter12
26
the Operational Sense, Attributes, Units, and Measurement Sense, and Patterns and
Relationships sub-strands (see Table 2).
All of the Junior assessments achieved acceptable levels of categorical
concurrence on all of the strands (see Tables A1.6 to A1.10 in Appendix 1). Only two of
the sub-strands had acceptable categorical concurrence on any of the Junior assessments:
the Measurement Relationships and the Patterns and Relationships sub-strands (see Table
3).
The main reason why categorical concurrence was rarely achieved on the sub-
strand level is simply that there were not enough assessment items to adequately measure
each of the thirteen sub-strands. Although multiple expectations were matched to some
questions so that the number of hits was greater than thirty-six, there were not enough
hits to adequately cover all of the sub-strands (see Table 4).
Range-of-Knowledge Correspondence
No Primary assessment met the range-of-knowledge criterion for all strands; the
number of strands for which the criterion was met ranged from one to four (see Tables
A1.11 to A1.15 in Appendix 1). Only one primary-level assessment met this criterion for
either the Measurement or the Geometry and Spatial Sense strands while all but one met
the criterion for the Number Sense and Numeration strand (Table 2). Satisfactory range-
of-knowledge was attained unevenly at the sub-strand level (see Table 3 for details).
No Junior assessment met the range-of-knowledge criterion for all strands (see
Tables A1.16 to A1.20 in Appendix 1). Two of the assessments attained a satisfactory
range-of-knowledge on 3 of 5 strands, failing to meet the criterion on the Number Sense
and Numeration and Geometry and Spatial Sense strands. The remaining three
ANALYSIS OF EQAO MATHEMATICS ASSESSMENTS
Mayes_04.20 EDUC783-Winter12
27
assessments attained a satisfactory level on 4 of the 5 strands, only failing to meet the
criterion on the Number Sense and Numeration strand. See Table 3 for information about
range-of-knowledge attainment at the sub-strand level.
Balance of Representation
All assessments studied met the balance of representation criterion for all strands
and sub-strands. Balance indices ranged from 0.71 to 1 on the Primary assessments and
from 0.73 to 1 on the Junior assessments; see Tables A1.11 to A1.20 in the Appendix for
details.
Comparison of curricula defined by EQAO tests, The Ontario Curriculum, and
textbooks
A summary of the strand-by-strand content from the 2007 to 2011 Primary and
Junior assessments is shown in Figures A2.1 and A2.2 in Appendix 2. These charts also
display the content of the standards (as measured by number of expectations per strand)
and the content of the textbooks studied (as measured by number of sections per strand).
On the strand level, the content of each of the primary-level assessments was
closely aligned to both the textbook examined and to the grade 3 curriculum (Tables A2.1
and A2.2). The most notable differences in content were in the Number Sense and
Numeration strand, which was emphasized more on four of the primary tests than in the
curriculum, and in the Measurement strand, which was emphasized less on three of the
primary tests than in the textbook.
The content of each of the junior-level assessments was closely aligned with the
grade 6 curriculum on the strand level (see Table A2.3). There were no consistent
differences between content of the tests and the curriculum. There were, however,
ANALYSIS OF EQAO MATHEMATICS ASSESSMENTS
Mayes_04.20 EDUC783-Winter12
28
significant differences between the content of the assessments and the grade 6 textbooks
examined (see Table A2.4; chi-square values ranged from 14.806 to 27.125 which were
significant at p=0.01). The primary discrepancy between the texts and the assessments
occurred in the Number Sense and Numeration strand: the textbooks emphasized this
strand far more than the assessments.
Mathematical Integrity of Assessment Items
Several of the assessment items use imprecise language or unclear instructions to
an extent that the question is either mathematically incorrect or it could be interpreted in
ways other than those considered in the Scoring Guide. Table 4 contains a summary of
such problematic items; I will explain the content of this table below. Overall, 18 out of
the 360 problems examined, or about 5%, had problematic mathematical content.
Table 4
Summary of assessment items with problematic content
Assessment Question Category of Concern
Area of Concern
1. 2007 Primary 7 Imprecise wording
All spinners could be generate the shown data
2. 9 Unclear instructions
Necessary ‘justification’ is unclear (see Scoring Guide)
3. 28 Pattern Cannot just extend pattern based on that shown in table
4. 2008 Primary 20 Imprecise wording
Confusing and imprecise wording
5. 30 Unclear instructions
Is not clear how many crayons are in each box
6. 2009 Primary 5 Pattern No description of rule to extend pattern 7. 7 Pattern Geometric situation unclear;
incomplete pattern rule 8. 29 Imprecise
wording Multiple shortest paths are possible; asks for “the” path
9. 2010 Primary 2 Pattern No description of rule to extend pattern 10. 2011 Primary 1 Pattern No description of rule to extend pattern 11. 9 Unclear
instructions Meaning of “on the grid lines” unclear
ANALYSIS OF EQAO MATHEMATICS ASSESSMENTS
Mayes_04.20 EDUC783-Winter12
29
12. 2008 Junior 2 Imprecise wording
Should state “In what time could Joseph’s friend…”
13. 6 Pattern No description of rule to extend pattern 14. 28 Significant
mathematical error
Insufficient information given to be able to answer question; incorrect answers accepted (see Scoring Guide)
15. 2010 Junior 15 Pattern No description of rule to extend pattern 16. 26 Pattern No description of rule to extend pattern 17. 2011 Junior 1 Pattern No description of rule to extend pattern 18. 25 Pattern No description of rule to extend pattern
Small differences in wording are important in mathematics; this precision was lost
in some of the assessment questions, making statements incorrect. For example, question
29 of the 2009 Primary Assessment asks students to "draw the shortest path he can take
from school to the park and then to his house" when there is more than one shortest path
that satisfies the given conditions (see Figure A3.1). This type of mistake may confuse
students who see that there are multiple correct answers to the problem. Another
example of imprecise wording is found in question 7 on the 2007 Primary Assessment
which asks which spinner "could be" used when, in fact, all of the spinners could produce
the data in the table (see Figure A3.2); this question should instead ask which spinner is
most likely to produce the data in the table. A third example of imprecise wording is
found in question 2 of the 2008 Junior Assessment which asks "in what time does
Joseph's friend swim the race" when there are multiple times possible (see Figure A3.3);
the question should instead ask “which of the following could be the time that Joseph’s
friend swims the race.”
Other questions are problematic because the instructions or conditions in the
problem are vague. Although question 9 on the 2011 Primary Assessment asks students
to draw the shortest path "on the grid lines" it is not clear whether this refers to the actual
ANALYSIS OF EQAO MATHEMATICS ASSESSMENTS
Mayes_04.20 EDUC783-Winter12
30
lines themselves or the entire area of the grid (see Figure A3.4); the Scoring Guide
reveals that students were required to draw a path that stays along the lines. Question 7
on the 2009 Primary Assessment simply states that "Sally is making triangles using
straws" but provides no information about how she is making those triangles or whether
she keeps making triangles in the same way (see Figure A3.5). The triangles may either
be disjoint, as the test writers assumed when writing this question, or they may share
sides; in fact, the corresponding expectation on forming sequences from geometric
sequences in the curriculum includes a similar problem where the shapes are assumed to
share sides. Other vague questions include problem 28 on the 2007 Primary Assessment
and question 30 on the 2008 Primary Assessment.
Short answer questions usually require students to justify their answer but what is
expected of a justification is unclear and at times uneven. For example, both question 9
of the 2007 Primary Assessment and question 25 of the 2010 Primary Assessment asks
students to find lines of symmetry and to "justify your answer." According to the scoring
guides, students who wrote the test in 2010 only had to show lines of symmetry while
students who wrote the test in 2007 were also required to give a definition of a line of
symmetry. This is problematic for teachers who look to scoring guides from past tests to
prepare their students for EQAO assessments.
Even students who had mastered the grade 6 curriculum would have a very
difficult time giving a correct answer to question 28 on the 2008 Junior Assessment (see
Figure A3.6) and, in fact, the example given in the Scoring Guide for demonstrating “a
complete solution process” is not complete. According to the scoring guide, a student
who wrote out the results of the rule for polygons with 3 to 8 sides with no justification
ANALYSIS OF EQAO MATHEMATICS ASSESSMENTS
Mayes_04.20 EDUC783-Winter12
31
related to the geometry was given full credit while a student who gave it only for
quadrilaterals is not. Each of these solutions is equally incorrect mathematically; it seems
as though the test writers are willing to perpetuate the misunderstanding that showing
'enough' examples constitutes a proof.
There is a second difficulty with Question 28 of the 2008 Junior Assessment: it
assumes that patterns are determined by their first few terms. While this is true when you
restrict your attention to certain types of patterns, such as those formed by geometric or
arithmetic sequences, not all patterns are of this sort; in fact one could make many
reasonable patterns out of any finite number sequence. Several questions on the
assessments studied make the same incorrect assumption. For example, question 25 on
the 2011 Junior Assessment assumes that because two tokens are earned per task for up
to five tasks that two tokens will be earned per task for other numbers of tasks (see Figure
A3.7). It is clear, however, that many other situations are possible under the conditions
given: for example, Cole might earn a bonus for every eight tasks completed or he may
only be paid for the first 5 tasks that he completes, earning 10 tokens for completing 5 or
greater tasks. See Table 4 for references to other questions with similar problems.
Discussion
EQAO tests are designed to follow The Ontario Curriculum and it is clear that the
writers pay careful attention to making sure that all strands are represented on the tests.
However, a deeper analysis of alignment, attending to sub-strands and to specific
standards, reveals problems. Further, there are difficulties with the language or
mathematics on about 5 percent of assessment items studied.
The EQAO assessments evaluated are very well matched to the curriculum at a
ANALYSIS OF EQAO MATHEMATICS ASSESSMENTS
Mayes_04.20 EDUC783-Winter12
32
strand level. As the comparison of textbook, curriculum, and assessment content showed,
the content of the assessments is much more closely aligned to curriculum than it is to the
textbooks. They test content from the curriculum at an appropriate grade level and do not
contain questions that are either too basic or too advanced for students of a particular age.
Further, assessments consistently met the categorical concurrence criterion at the strand
level, demonstrating that the EQAO assessments are able to measure students’ mastery of
each of the strands. In a similar vein, the EQAO assessments are well-balanced, in that
they do not significantly emphasize one expectation over another from year-to-year.
These observations indicate that the assessments are truly based on the Ontario
curriculum.
None of the assessments met the categorical concurrence criterion on the sub-
strand level or the range of knowledge criterion on either the strand or the sub-strand
level. It can be argued that this is primarily due to the length of the assessments: there
are thirteen sub-strands subdivided into 66 expectations in the grade 3 curriculum and 61
expectations in the grade 6 curriculum and only 36 assessment items on each test. A test
with 36 items based on The Ontario Curriculum would need to have more than two
expectations corresponding to every single item to have a shot at meeting both of these
alignment criteria. Even under these conditions, the objectives would have to be
perfectly balanced throughout the curriculum. Therefore, if the assessments are to truly
measure mastery of the breadth of the primary and junior curricula, it is necessary to
either create assessments with a greater number of questions or to design more items that
touch on several areas of the curricula.
The structure and the testing conditions of the EQAO assessments limit which
ANALYSIS OF EQAO MATHEMATICS ASSESSMENTS
Mayes_04.20 EDUC783-Winter12
33
expectations and aspects of expectations can be authentically measured and limits the
ability for the tests to meet the range of knowledge criterion. A written assessment
completed individually in a classroom over a short amount of time simply cannot
measure the mastery of certain expectations, such as those involving actual collection of
data, ability to do mental arithmetic, or explorations through experiments. Detecting true
understanding of even seemingly straightforward expectations, such as understanding the
meaning of the 'whole' in a fraction, can be impossible without a more interactive form of
assessment (Ball & Peoples, 2007). Further, the fact that students have access to
calculators throughout the entire Junior Assessment means that none of the computational
fluency objectives at this level can be measured.
Attempting to address some of these difficult-to-measure expectations has
resulted in some awkward EQAO assessment items. A better solution would be to
modify the testing conditions by limiting calculator use on the grade 6 assessment,
including oral questions to test mental arithmetic, and adding an interview component to
more accurately assess conceptual understanding.10 Clearly, some of these changes are
more feasible than others; where changes are not possible, it should be acknowledged that
certain parts of the curriculum cannot be honestly assessed by the EQAO tests.
Five percent of the EQAO assessment items evaluated were not entirely
mathematically sound. Precision is one of the most important aspects of mathematics and
is a key characteristic that sets it apart from other disciplines; even if these errors do not
impact overall scores, allowing precision to slip on such a high-profile assessment is
unacceptable. Even the youngest students deserve an intellectually honest presentation of
10 Writing multiple tests to be given to different groups of students will not solve the alignment problem since individual progress of students is also tracked.
ANALYSIS OF EQAO MATHEMATICS ASSESSMENTS
Mayes_04.20 EDUC783-Winter12
34
each subject that they study and the vague definitions, incorrect statements, and unclear
instructions that have appeared on the assessments have no place in mathematics. To
ensure that these issues do not continue to appear, trained mathematicians must be
meaningfully involved in every stage of the EQAO testing process, from drafting and
editing test questions to grading students' responses.11
Assessment design as much an art as it is a science and it is exceedingly difficult
to craft perfect assessments. However, given the increasing prominence and importance
of standards-based assessments, it is incumbent on organizations such as the EQAO to
write the best tests possible. This report has highlighted some aspects of Ontario's large-
scale elementary assessment that are commendable and others that can be improved with
changes to the structure of the tests, to the conditions under which the tests are taken, and
to the assessment writing and grading process. With these adjustments, the EQAO will
be able to provide more accurate information and send clearer messages to students,
teachers, and the public and will improve education accountability in Ontario.
11 I could not find any evidence that mathematicians are involved in the writing of the EQAO assessments.
ANALYSIS OF EQAO MATHEMATICS ASSESSMENTS
Mayes_04.20 EDUC783-Winter12
35
References
Ball, D. L. & Peoples, B. (2007). Assessing a student’s mathematical knowledge by way
of interview. In A. Schoenfeld (Ed.), Assessing Mathematical Proficiency. New
York: Cambridge University Press.
Chandler, D. G., & Bronsan, P. A. (1995). A comparison between mathematics textbook
content and a statewide mathematics proficiency test. School Science and
Mathematics, 95(3), 118-123.
Education Quality and Accountability Office (2012). The Power of Ontario’s Provincial
Testing Program. Toronto: Queen’s Printer for Ontario.
Farenga, S. J., Joyce, B. A., & Ness, D. (2002). Reaching the zone of optimal learning:
The alignment of curriculum, instruction, and assessment. In R.W. Bybee (Ed.),
Learning science and the science of learning. Arlington, VA: NSTA Press.
Herman, J. L. (2004). The effects of testing on instruction. In S. Furhman & R. F.
Elmore (Eds.), Redesigning accountability systems for education (pp. 141-166).
New York: Teachers College Press.
Herman, J.L., Webb, N. L., Zuniga, S. A. (2007). Measurement issues in the alignment
of standards and assessments: A case study. Journal of Applied Measurement in
Education, 20(1), 101-126.
Kozlow, M. (2007). Model Selection for the Analysis of EQAO Assessment Data.
EQAO Research Bulletin #1. Retrieved from
http://www.eqao.com/Research/pdf/E/1_Research_Bulletin_1207_web.pdf.
La Marca, P. M. (2001). Alignment of standards and assessments as accountability
criterion. Practical Assessment, Research & Evaluation, 7(21).
ANALYSIS OF EQAO MATHEMATICS ASSESSMENTS
Mayes_04.20 EDUC783-Winter12
36
Mohamud, A. & Fleck, D. (2010). Alignment of standards, assessment, and instruction:
Implications for English Language Learners in Ohio. Theory Into Practice, 49(2),
129-136.
Näsström, G. & Henriksson, W. (2008). Alignment of standards and assessments: a
theoretical and empirical study of methods for alignment. Electronic Journal for
Research in Educational Psychology, 16(6), 667-690.
Ontario Ministry of Education (2005). The Ontario Curriculum, Grades 1-8:
Mathematics. Toronto: Queen’s Printer for Ontario.
Porter, A. C. (2002). Measuring the content of instruction: Uses in research and practice
(AERA 2002 Presidential Address). Educational Researcher, 31(7), 3-14.
Roach, A. T., Elliott, S. N., & Webb, N. L. (2005). Alignment of an alternate assessment
with state academic standards: The validity of the Wisconsin Alternate Assessment.
Journal of Special Education, 38(4), 218-231.
Rothman, R. (2003). Imperfect matches: The alignment of standards and tests. Paper
commissioned by the Committee on Test Design for K-12 Science Achievement,
March 2003.
Webb, N.L. (1999). Alignment of science and mathematics standards and assessments in
four states (Research monograph, No. 18). Madison: National Institute for Science
Education.
Webb, N. L., 2005. Web Alignment Tool (WAT) Training Manual. Retrieved February 12,
2012. Available: http://www.wcer.wisc.edu/wat/index.aspx.
ANALYSIS OF EQAO MATHEMATICS ASSESSMENTS
Mayes_04.20 EDUC783-Winter12
37
Wixson, K. K., Fisk, M. C., Dutro, E., & McDaniel, J. (2002). The alignment of state
standards and assessments in elementary reading. CIERA Report #3-024. Ann
Arbor: University of Michigan, School of Education, Center for the Improvement
of Early Reading Achievement.
ANALYSIS OF EQAO MATHEMATICS ASSESSMENTS
Mayes_04.20 EDUC783-Winter12
38
Appendix 1
Detailed Results of Alignment Analyses
Table A1.1
Categorical Concurrence: Primary Assessment 2007
Strands and Sub-Strands Expect. # a
Hitsb Categorical Concurr.
Acceptablec
NSN: Number Sense and Numeration 19 18 ● NSN1: Quantity Relationships 10 3 ❍
NSN2: Counting 2 3 ❍ NSN3: Operational Sense 7d 12 ●
M: Measurement 16 11 ● M1: Attributes, Units, and Meas. Sense 10 6 ●
M2: Measurement Relationships 6 5 ❍ GSS: Geometry and Spatial Sense 13 5 ❍
GSS1: Geometric Properties 5 1 ❍ GSS2: Geometric Relationships 5 2 ❍ GSS3: Location and Movement 3 2 ❍
PA: Patterning and Algebra 10 6 ● PA1: Patterns and Relationships 6 3 ❍
PA2: Expressions and Equality 4 3 ❍ DMP: Data Management and Probability 8 6 ●
DMP1: Collection and Organization of Data 3 2 ❍ DMP2: Data Relationships 3 1 ❍
DMP3: Probability 2 3 ❍ a Expect. # refers to the number of expectations in a given strand or sub-strand in The Ontario Curriculum for grade 3 b Hits refers to the number of items that were coded as corresponding to an expectation within each strand or sub-strand c Categorical Concurr. Acceptable: ● indicates that the strand or sub-strand met the acceptable level for categorical concurrence on a given assessment (at least 6 hits); ❍ indicates that the criterion was not met. d Includes one additional expectation because coded items did not correspond to existing expectations (see the Methods section).
ANALYSIS OF EQAO MATHEMATICS ASSESSMENTS
Mayes_04.20 EDUC783-Winter12
39
Table A1.2
Categorical Concurrence: Primary Assessment 2008
Strands and Sub-Strands Expect.
#a Hitsb Categorical
Concurr. Acceptablec
NSN: Number Sense and Numeration 19 16 ● NSN1: Quantity Relationships 10 5 ❍
NSN2: Counting 2 3 ❍ NSN3: Operational Sense 7d 8 ●
M: Measurement 16 9 ● M1: Attributes, Units, and Meas. Sense 10 6 ●
M2: Measurement Relationships 6 3 ❍ GSS: Geometry and Spatial Sense 13 7 ●
GSS1: Geometric Properties 5 4 ❍ GSS2: Geometric Relationships 5 1 ❍ GSS3: Location and Movement 3 2 ❍
PA: Patterning and Algebra 10 7 ● PA1: Patterns and Relationships 6 6 ●
PA2: Expressions and Equality 4 1 ❍ DMP: Data Management and Probability 8 5 ❍
DMP1: Collection and Organization of Data 3 2 ❍ DMP2: Data Relationships 3 1 ❍
DMP3: Probability 2 2 ❍ a Expect. # refers to the number of expectations in a given strand or sub-strand in The Ontario Curriculum for grade 3 b Hits refers to the number of items that were coded as corresponding to an expectation within each strand or sub-strand c Categorical Concurr. Acceptable: ● indicates that the strand or sub-strand met the acceptable level for categorical concurrence on a given assessment (at least 6 hits); ❍ indicates that the criterion was not met. d Includes one additional expectation because coded items did not correspond to existing expectations (see the Methods section).
ANALYSIS OF EQAO MATHEMATICS ASSESSMENTS
Mayes_04.20 EDUC783-Winter12
40
Table A1.3
Categorical Concurrence: Primary Assessment 2009
Strands and Sub-Strands Expect.
#a Hitsb Categorical
Concurr. Acceptablec
NSN: Number Sense and Numeration 19 20 ● NSN1: Quantity Relationships 10 3 ❍
NSN2: Counting 2 3 ❍ NSN3: Operational Sense 7d 14 ●
M: Measurement 16 8 ● M1: Attributes, Units, and Meas. Sense 10 5 ❍
M2: Measurement Relationships 6 3 ❍ GSS: Geometry and Spatial Sense 13 4 ❍
GSS1: Geometric Properties 5 1 ❍ GSS2: Geometric Relationships 5 2 ❍ GSS3: Location and Movement 3 1 ❍
PA: Patterning and Algebra 10 6 ● PA1: Patterns and Relationships 6 3 ❍
PA2: Expressions and Equality 4 3 ❍ DMP: Data Management and Probability 8 5 ❍
DMP1: Collection and Organization of Data 3 2 ❍ DMP2: Data Relationships 3 2 ❍
DMP3: Probability 2 1 ❍ a Expect. # refers to the number of expectations in a given strand or sub-strand in The Ontario Curriculum for grade 3 b Hits refers to the number of items that were coded as corresponding to an expectation within each strand or sub-strand c Categorical Concurr. Acceptable: ● indicates that the strand or sub-strand met the acceptable level for categorical concurrence on a given assessment (at least 6 hits); ❍ indicates that the criterion was not met. d Includes one additional expectation because coded items did not correspond to existing expectations (see the Methods section).
ANALYSIS OF EQAO MATHEMATICS ASSESSMENTS
Mayes_04.20 EDUC783-Winter12
41
Table A1.4
Categorical Concurrence: Primary Assessment 2010
Strands and Sub-Strands Expect.
#a Hitsb Categorical
Concurr. Acceptablec
NSN: Number Sense and Numeration 19 16 ● NSN1: Quantity Relationships 10 4 ❍
NSN2: Counting 2 2 ❍ NSN3: Operational Sense 7d 10 ●
M: Measurement 16 9 ● M1: Attributes, Units, and Meas. Sense 10 5 ❍
M2: Measurement Relationships 6 4 ❍ GSS: Geometry and Spatial Sense 13 7 ●
GSS1: Geometric Properties 5 1 ❍ GSS2: Geometric Relationships 5 4 ❍ GSS3: Location and Movement 3 2 ❍
PA: Patterning and Algebra 10 7 ● PA1: Patterns and Relationships 6 5 ❍
PA2: Expressions and Equality 4 2 ❍ DMP: Data Management and Probability 8 4 ❍
DMP1: Collection and Organization of Data 3 2 ❍ DMP2: Data Relationships 3 1 ❍
DMP3: Probability 2 1 ❍ a Expect. # refers to the number of expectations in a given strand or sub-strand in The Ontario Curriculum for grade 3 b Hits refers to the number of items that were coded as corresponding to an expectation within each strand or sub-strand c Categorical Concurr. Acceptable: ● indicates that the strand or sub-strand met the acceptable level for categorical concurrence on a given assessment (at least 6 hits); ❍ indicates that the criterion was not met. d Includes one additional expectation because coded items did not correspond to existing expectations (see the Methods section).
ANALYSIS OF EQAO MATHEMATICS ASSESSMENTS
Mayes_04.20 EDUC783-Winter12
42
Table A1.5 Categorical Concurrence: Primary Assessment 2011
Strands and Sub-Strands Expect.
#a Hitsb Categorical
Concurr. Acceptablec
NSN: Number Sense and Numeration 19 21 ● NSN1: Quantity Relationships 10 2 ❍
NSN2: Counting 2 1 ❍ NSN3: Operational Sense 7d 18 ●
M: Measurement 16 8 ● M1: Attributes, Units, and Meas. Sense 10 6 ●
M2: Measurement Relationships 6 2 ❍ GSS: Geometry and Spatial Sense 13 7 ●
GSS1: Geometric Properties 5 1 ❍ GSS2: Geometric Relationships 5 4 ❍ GSS3: Location and Movement 3 2 ❍
PA: Patterning and Algebra 10 7 ● PA1: Patterns and Relationships 6 6 ●
PA2: Expressions and Equality 4 1 ❍ DMP: Data Management and Probability 8 6 ●
DMP1: Collection and Organization of Data 3 2 ❍ DMP2: Data Relationships 3 2 ❍
DMP3: Probability 2 2 ❍ a Expect. # refers to the number of expectations in a given strand or sub-strand in The Ontario Curriculum for grade 3 b Hits refers to the number of items that were coded as corresponding to an expectation within each strand or sub-strand c Categorical Concurr. Acceptable: ● indicates that the strand or sub-strand met the acceptable level for categorical concurrence on a given assessment (at least 6 hits); ❍ indicates that the criterion was not met. d Includes one additional expectation because coded items did not correspond to existing expectations (see the Methods section).
ANALYSIS OF EQAO MATHEMATICS ASSESSMENTS
Mayes_04.20 EDUC783-Winter12
43
Table A1.6 Categorical Concurrence: Junior Assessment 2007
Strands and Sub-Strands Expect.
#a Hitsb Categorical
Concurr. Acceptablec
NSN: Number Sense and Numeration 18 8 ● NSN1: Quantity Relationships 7 2 ❍
NSN3: Operational Sense 8 2 ❍ NSN4: Proportional Relationships 3 4 ❍
M: Measurement 12 10 ● M1: Attributes, Units, and Meas. Sense 2 2 ❍
M2: Measurement Relationships 10 8 ● GSS: Geometry and Spatial Sense 9 8 ●
GSS1: Geometric Properties 4 3 ❍ GSS2: Geometric Relationships 2 1 ❍ GSS3: Location and Movement 3 4 ❍
PA: Patterning and Algebra 10 7 ● PA1: Patterns and Relationships 6 2 ❍
PA2: Variables, Expressions, and Equations 4 5 ❍ DMP: Data Management and Probability 12 7 ●
DMP1: Collection and Organization of Data 4 0 ❍ DMP2: Data Relationships 5 4 ❍
DMP3: Probability 3 3 ❍ a Expect. # refers to the number of expectations in a given strand or sub-strand in The Ontario Curriculum for grade 6 b Hits refers to the number of items that were coded as corresponding to an expectation within each strand or sub-strand c Categorical Concurr. Acceptable: ● indicates that the strand or sub-strand met the acceptable level for categorical concurrence on a given assessment (at least 6 hits); ❍ indicates that the criterion was not met.
ANALYSIS OF EQAO MATHEMATICS ASSESSMENTS
Mayes_04.20 EDUC783-Winter12
44
Table A1.7
Categorical Concurrence: Junior Assessment 2008
Strands and Sub-Strands Expect.
#a Hitsb Categorical
Concurr. Acceptablec
NSN: Number Sense and Numeration 18 9 ● NSN1: Quantity Relationships 7 4 ❍
NSN3: Operational Sense 8 2 ❍ NSN4: Proportional Relationships 3 3 ❍
M: Measurement 12 8 ● M1: Attributes, Units, and Meas. Sense 2 1 ❍
M2: Measurement Relationships 10 7 ● GSS: Geometry and Spatial Sense 9 7 ●
GSS1: Geometric Properties 4 2 ❍ GSS2: Geometric Relationships 2 1 ❍ GSS3: Location and Movement 3 4 ❍
PA: Patterning and Algebra 10 11 ● PA1: Patterns and Relationships 6 8 ●
PA2: Variables, Expressions, and Equations 4 3 ❍ DMP: Data Management and Probability 12 7 ●
DMP1: Collection and Organization of Data 4 1 ❍ DMP2: Data Relationships 5 3 ❍
DMP3: Probability 3 3 ❍ a Expect. # refers to the number of expectations in a given strand or sub-strand in The Ontario Curriculum for grade 6 b Hits refers to the number of items that were coded as corresponding to an expectation within each strand or sub-strand c Categorical Concurr. Acceptable: ● indicates that the strand or sub-strand met the acceptable level for categorical concurrence on a given assessment (at least 6 hits); ❍ indicates that the criterion was not met.
ANALYSIS OF EQAO MATHEMATICS ASSESSMENTS
Mayes_04.20 EDUC783-Winter12
45
Table A1.8
Categorical Concurrence: Junior Assessment 2009
Strands and Sub-Strands Expect.
#a Hitsb Categorical
Concurr. Acceptablec
NSN: Number Sense and Numeration 18 9 ● NSN1: Quantity Relationships 7 3 ❍
NSN3: Operational Sense 8 3 ❍ NSN4: Proportional Relationships 3 3 ❍
M: Measurement 12 9 ● M1: Attributes, Units, and Meas. Sense 2 0 ❍
M2: Measurement Relationships 10 9 ● GSS: Geometry and Spatial Sense 9 8 ●
GSS1: Geometric Properties 4 3 ❍ GSS2: Geometric Relationships 2 0 ❍ GSS3: Location and Movement 3 5 ❍
PA: Patterning and Algebra 10 6 ● PA1: Patterns and Relationships 6 4 ❍
PA2: Variables, Expressions, and Equations 4 2 ❍ DMP: Data Management and Probability 12 8 ●
DMP1: Collection and Organization of Data 4 3 ❍ DMP2: Data Relationships 5 2 ❍
DMP3: Probability 3 3 ❍ a Expect. # refers to the number of expectations in a given strand or sub-strand in The Ontario Curriculum for grade 6 b Hits refers to the number of items that were coded as corresponding to an expectation within each strand or sub-strand c Categorical Concurr. Acceptable: ● indicates that the strand or sub-strand met the acceptable level for categorical concurrence on a given assessment (at least 6 hits); ❍ indicates that the criterion was not met.
ANALYSIS OF EQAO MATHEMATICS ASSESSMENTS
Mayes_04.20 EDUC783-Winter12
46
Table A1.9
Categorical Concurrence: Junior Assessment 2010
Strands and Sub-Strands Expect.
#a Hitsb Categorical
Concurr. Acceptablec
NSN: Number Sense and Numeration 18 9 ● NSN1: Quantity Relationships 7 2 ❍
NSN3: Operational Sense 8 3 ❍ NSN4: Proportional Relationships 3 4 ❍
M: Measurement 12 10 ● M1: Attributes, Units, and Meas. Sense 2 1 ❍
M2: Measurement Relationships 10 9 ● GSS: Geometry and Spatial Sense 9 7 ●
GSS1: Geometric Properties 4 2 ❍ GSS2: Geometric Relationships 2 1 ❍ GSS3: Location and Movement 3 4 ❍
PA: Patterning and Algebra 10 7 ● PA1: Patterns and Relationships 6 6 ●
PA2: Variables, Expressions, and Equations 4 1 ❍ DMP: Data Management and Probability 12 6 ●
DMP1: Collection and Organization of Data 4 0 ❍ DMP2: Data Relationships 5 4 ❍
DMP3: Probability 3 2 ❍ a Expect. # refers to the number of expectations in a given strand or sub-strand in The Ontario Curriculum for grade 6 b Hits refers to the number of items that were coded as corresponding to an expectation within each strand or sub-strand c Categorical Concurr. Acceptable: ● indicates that the strand or sub-strand met the acceptable level for categorical concurrence on a given assessment (at least 6 hits); ❍ indicates that the criterion was not met.
ANALYSIS OF EQAO MATHEMATICS ASSESSMENTS
Mayes_04.20 EDUC783-Winter12
47
Table A1.10
Categorical Concurrence: Junior Assessment 2011
Strands and Sub-Strands Expect.
#a Hitsb Categorical
Concurr. Acceptablec
NSN: Number Sense and Numeration 18 9 ● NSN1: Quantity Relationships 7 2 ❍
NSN3: Operational Sense 8 5 ❍ NSN4: Proportional Relationships 3 2 ❍
M: Measurement 12 8 ● M1: Attributes, Units, and Meas. Sense 2 1 ❍
M2: Measurement Relationships 10 7 ● GSS: Geometry and Spatial Sense 9 7 ●
GSS1: Geometric Properties 4 3 ❍ GSS2: Geometric Relationships 2 1 ❍ GSS3: Location and Movement 3 3 ❍
PA: Patterning and Algebra 10 8 ● PA1: Patterns and Relationships 6 5 ❍
PA2: Variables, Expressions, and Equations 4 3 ❍ DMP: Data Management and Probability 12 9 ●
DMP1: Collection and Organization of Data 4 2 ❍ DMP2: Data Relationships 5 3 ❍
DMP3: Probability 3 4 ❍ a Expect. # refers to the number of expectations in a given strand or sub-strand in The Ontario Curriculum for grade 6 b Hits refers to the number of items that were coded as corresponding to an expectation within each strand or sub-strand c Categorical Concurr. Acceptable: ● indicates that the strand or sub-strand met the acceptable level for categorical concurrence on a given assessment (at least 6 hits); ❍ indicates that the criterion was not met.
ANALYSIS OF EQAO MATHEMATICS ASSESSMENTS
Mayes_04.20 EDUC783-Winter12
48
Table A1.11
Range-of-Knowledge Correspondence and Balance of Representation: Primary Assessment 2007
Strands and Sub-Strands Range of Objectives Range of
Know. Accept.c
Balance Indexd
Balance of Rep.
Accept.e Title Expect. # # Hita % of
Totalb
NSN1: Number Sense and Numeration 19 9 50 ● 0.83 ● NSN1: Quantity Relationships 10 2 20 ❍ 0.83 ●
NSN2: Counting 2 2 100 ● 0.83 ● NSN3: Operational Sense 7 5 71 ● 0.82 ●
M: Measurement 16 9 56 ● 0.94 ● M1: Attributes, Units, and Meas. Sense 10 4 40 ❍ 0.83 ●
M2: Measurement Relationships 6 5 83 ● 1 ● GSS: Geometry and Spatial Sense 13 4 31 ❍ 0.85 ●
GSS1: Geometric Properties 5 1 20 ❍ 1 ● GSS2: Geometric Relationships 5 1 20 ❍ 1 ● GSS3: Location and Movement 3 2 66 ● 1 ●
PA: Patterning and Algebra 10 5 50 ● 0.87 ● PA1: Patterns and Relationships 6 3 50 ● 1 ●
PA2: Expressions and Equality 4 2 50 ● 0.83 ● DMP: Data Management and Probability 8 4 50 ● 0.75 ● DMP1: Collection and Organization of Data 3 2 66 ● 1 ●
DMP2: Data Relationships 3 1 33 ❍ 1 ● DMP3: Probability 2 1 50 ● 1 ●
a # Hit: The number of expectations within a strand or a sub-strand that were hit at least once by items on the assessment. b % of Total: The percentage of expectations within a strand or a sub-strand that were hit by items on the assessment. c Range of Know. Accept.: ● indicates that 50% or more of the expectations within a strand or a sub-strand were hit to satisfy the Range-of-Knowledge criterion; ❍ indicates that this criterion was not met. d Balance Index: The balance index for the expectations hit within a strand or a sub-strand. e Balance of Rep. Accept.: ● indicates that the balance index for the strand or sub-strand was high enough to meet the Balance of Representation criterion (0.7 or above); ❍ indicates that the criterion was not met.
ANALYSIS OF EQAO MATHEMATICS ASSESSMENTS
Mayes_04.20 EDUC783-Winter12
49
Table A1.12 Range-of-Knowledge Correspondence and Balance of Representation: Primary Assessment 2008
Strands and Sub-Strands Range of Objectives Range of
Know. Accept.c
Balance Indexd
Balance of Rep.
Accept.e Title Expect. # # Hita % of
Totalb
NSN1: Number Sense and Numeration 19 11 58 ● 0.80 ● NSN1: Quantity Relationships 10 4 40 ❍ 0.85 ●
NSN2: Counting 2 2 100 ● 0.83 ● NSN3: Operational Sense 7 5 71 ● 0.78 ●
M: Measurement 16 7 44 ❍ 0.84 ● M1: Attributes, Units, and Meas. Sense 10 4 40 ❍ 0.83 ●
M2: Measurement Relationships 6 3 50 ● 1 ● GSS: Geometry and Spatial Sense 13 7 54 ● 1 ●
GSS1: Geometric Properties 5 4 80 ● 1 ● GSS2: Geometric Relationships 5 1 20 ❍ 1 ● GSS3: Location and Movement 3 2 66 ● 1 ●
PA: Patterning and Algebra 10 5 50 ● 0.83 ● PA1: Patterns and Relationships 6 4 66 ● 0.83 ●
PA2: Expressions and Equality 4 1 25 ❍ 1 ● DMP: Data Management and Probability 8 3 38 ❍ 0.87 ● DMP1: Collection and Organization of Data 3 1 33 ❍ 1 ●
DMP2: Data Relationships 3 1 33 ❍ 1 ● DMP3: Probability 2 1 50 ● 1 ●
a # Hit: The number of expectations within a strand or a sub-strand that were hit at least once by items on the assessment. b % of Total: The percentage of expectations within a strand or a sub-strand that were hit by items on the assessment. c Range of Know. Accept.: ● indicates that 50% or more of the expectations within a strand or a sub-strand were hit to satisfy the Range-of-Knowledge criterion; ❍ indicates that this criterion was not met. d Balance Index: The balance index for the expectations hit within a strand or a sub-strand. e Balance of Rep. Accept.: ● indicates that the balance index for the strand or sub-strand was high enough to meet the Balance of Representation criterion (0.7 or above); ❍ indicates that the criterion was not met.
ANALYSIS OF EQAO MATHEMATICS ASSESSMENTS
Mayes_04.20 EDUC783-Winter12
50
Table A1.13 Range-of-Knowledge Correspondence and Balance of Representation: Primary Assessment 2009
Strands and Sub-Strands Range of Objectives Range of
Know. Accept.c
Balance Indexd
Balance of Rep.
Accept.e Title Expect. # # Hita % of
Totalb
NSN1: Number Sense and Numeration 19 10 53 ● 0.75 ● NSN1: Quantity Relationships 10 2 20 ❍ 0.83 ●
NSN2: Counting 2 2 100 ● 0.83 ● NSN3: Operational Sense 7 6 86 ● 0.71 ●
M: Measurement 16 7 44 ❍ 0.89 ● M1: Attributes, Units, and Meas. Sense 10 4 40 ❍ 0.85 ●
M2: Measurement Relationships 6 3 50 ● 1 ● GSS: Geometry and Spatial Sense 13 4 31 ❍ 1 ●
GSS1: Geometric Properties 5 1 20 ❍ 1 ● GSS2: Geometric Relationships 5 2 40 ❍ 1 ● GSS3: Location and Movement 3 1 33 ❍ 1 ●
PA: Patterning and Algebra 10 5 50 ● 0.87 ● PA1: Patterns and Relationships 6 3 50 ● 1 ●
PA2: Expressions and Equality 4 2 50 ● 0.83 ● DMP: Data Management and Probability 8 4 50 ● 0.87 ● DMP1: Collection and Organization of Data 3 1 33 ❍ 1 ●
DMP2: Data Relationships 3 2 66 ● 1 ● DMP3: Probability 2 1 50 ● 1 ●
a # Hit: The number of expectations within a strand or a sub-strand that were hit at least once by items on the assessment. b % of Total: The percentage of expectations within a strand or a sub-strand that were hit by items on the assessment. c Range of Know. Accept.: ● indicates that 50% or more of the expectations within a strand or a sub-strand were hit to satisfy the Range-of-Knowledge criterion; ❍ indicates that this criterion was not met. d Balance Index: The balance index for the expectations hit within a strand or a sub-strand. e Balance of Rep. Accept.: ● indicates that the balance index for the strand or sub-strand was high enough to meet the Balance of Representation criterion (0.7 or above); ❍ indicates that the criterion was not met.
ANALYSIS OF EQAO MATHEMATICS ASSESSMENTS
Mayes_04.20 EDUC783-Winter12
51
Table A1.14
Range-of-Knowledge Correspondence and Balance of Representation: Primary Assessment 2010
Strands and Sub-Strands Range of Objectives Range of
Know. Accept.c
Balance Indexd
Balance of Rep.
Accept.e Title Expect. # # Hita % of
Totalb
NSN1: Number Sense and Numeration 19 10 53 ● 0.81 ● NSN1: Quantity Relationships 10 3 30 ❍ 0.83 ●
NSN2: Counting 2 1 50 ● 1 ● NSN3: Operational Sense 7 6 86 ● 0.8 ●
M: Measurement 16 7 44 ❍ 0.84 ● M1: Attributes, Units, and Meas. Sense 10 5 50 ● 1 ●
M2: Measurement Relationships 6 2 33 ❍ 1 ● GSS: Geometry and Spatial Sense 13 5 38 ❍ 0.83 ●
GSS1: Geometric Properties 5 1 20 ❍ 1 ● GSS2: Geometric Relationships 5 2 40 ❍ 1 ● GSS3: Location and Movement 3 2 66 ● 1 ●
PA: Patterning and Algebra 10 4 40 ❍ 0.79 ● PA1: Patterns and Relationships 6 2 33 ❍ 0.9 ●
PA2: Expressions and Equality 4 2 50 ● 1 ● DMP: Data Management and Probability 8 3 38 ❍ 0.83 ● DMP1: Collection and Organization of Data 3 1 33 ❍ 1 ●
DMP2: Data Relationships 3 1 33 ❍ 1 ● DMP3: Probability 2 1 50 ● 1 ●
a # Hit: The number of expectations within a strand or a sub-strand that were hit at least once by items on the assessment. b % of Total: The percentage of expectations within a strand or a sub-strand that were hit by items on the assessment. c Range of Know. Accept.: ● indicates that 50% or more of the expectations within a strand or a sub-strand were hit to satisfy the Range-of-Knowledge criterion; ❍ indicates that this criterion was not met. d Balance Index: The balance index for the expectations hit within a strand or a sub-strand. e Balance of Rep. Accept.: ● indicates that the balance index for the strand or sub-strand was high enough to meet the Balance of Representation criterion (0.7 or above); ❍ indicates that the criterion was not met.
ANALYSIS OF EQAO MATHEMATICS ASSESSMENTS
Mayes_04.20 EDUC783-Winter12
52
Table A1.15 Range-of-Knowledge Correspondence and Balance of Representation: Primary Assessment 2011
Strands and Sub-Strands Range of Objectives Range of
Know. Accept.c
Balance Indexd
Balance of Rep.
Accept.e Title Expect. # # Hita % of
Totalb
NSN1: Number Sense and Numeration 19 7 37 ❍ 0.71 ● NSN1: Quantity Relationships 10 1 10 ❍ 1 ●
NSN2: Counting 2 1 50 ● 1 ● NSN3: Operational Sense 7 5 71 ● 0.79 ●
M: Measurement 16 5 31 ❍ 0.76 ● M1: Attributes, Units, and Meas. Sense 10 4 40 ❍ 0.75 ●
M2: Measurement Relationships 6 1 17 ❍ 1 ● GSS: Geometry and Spatial Sense 13 5 38 ❍ 0.85 ●
GSS1: Geometric Properties 5 1 20 ❍ 1 ● GSS2: Geometric Relationships 5 2 40 ❍ 1 ● GSS3: Location and Movement 3 2 66 ● 1 ●
PA: Patterning and Algebra 10 5 50 ● 0.87 ● PA1: Patterns and Relationships 6 4 66 ● 0.85 ●
PA2: Expressions and Equality 4 1 25 ❍ 1 ● DMP: Data Management and Probability 8 5 63 ● 0.87 ● DMP1: Collection and Organization of Data 3 1 33 ❍ 1 ●
DMP2: Data Relationships 3 2 66 ● 1 ● DMP3: Probability 2 2 100 ● 1 ●
a # Hit: The number of expectations within a strand or a sub-strand that were hit at least once by items on the assessment. b % of Total: The percentage of expectations within a strand or a sub-strand that were hit by items on the assessment. c Range of Know. Accept.: ● indicates that 50% or more of the expectations within a strand or a sub-strand were hit to satisfy the Range-of-Knowledge criterion; ❍ indicates that this criterion was not met. d Balance Index: The balance index for the expectations hit within a strand or a sub-strand. e Balance of Rep. Accept.: ● indicates that the balance index for the strand or sub-strand was high enough to meet the Balance of Representation criterion (0.7 or above); ❍ indicates that the criterion was not met.
ANALYSIS OF EQAO MATHEMATICS ASSESSMENTS
Mayes_04.20 EDUC783-Winter12
53
Table A1.16 Range-of-Knowledge Correspondence and Balance of Representation: Junior Assessment 2007
Strands and Sub-Strands Range of Objectives Range of
Know. Accept.c
Balance Indexd
Balance of Rep.
Accept.e Title Expec. # # Hita % of
Totalb
NSN1: Number Sense and Numeration 18 6 33 ❍ 0.83 ● NSN1: Quantity Relationships 7 2 29 ❍ 1 ●
NSN3: Operational Sense 8 2 25 ❍ 1 ● NSN4: Proportional Relationships 3 2 67 ● 1 ●
M: Measurement 12 9 75 ● 0.91 ● M1: Attributes, Units, and Meas. Sense 2 2 100 ● 1 ●
M2: Measurement Relationships 10 7 70 ● 0.89 ● GSS: Geometry and Spatial Sense 9 6 67 ● 0.88 ●
GSS1: Geometric Properties 4 2 50 ● 0.83 ● GSS2: Geometric Relationships 2 1 50 ● 1 ● GSS3: Location and Movement 3 3 100 ● 0.88 ●
PA: Patterning and Algebra 10 5 50 ● 0.77 ● PA1: Patterns and Relationships 6 2 33 ❍ 1 ●
PA2: Variables, Expressions, and Equations 4 3 75 ● 0.73 ● DMP: Data Management and Probability 12 5 42 ❍ 0.77 ● DMP1: Collection and Organization of Data 4 0 0 ❍ N/A ●
DMP2: Data Relationships 5 4 80 ● 1 ● DMP3: Probability 3 1 33 ❍ 1 ●
a # Hit: The number of expectations within a strand or a sub-strand that were hit at least once by items on the assessment. b % of Total: The percentage of expectations within a strand or a sub-strand that were hit by items on the assessment. c Range of Know. Accept.: ● indicates that 50% or more of the expectations within a strand or a sub-strand were hit to satisfy the Range-of-Knowledge criterion; ❍ indicates that this criterion was not met. d Balance Index: The balance index for the expectations hit within a strand or a sub-strand. e Balance of Rep. Accept.: ● indicates that the balance index for the strand or sub-strand was high enough to meet the Balance of Representation criterion (0.7 or above); ❍ indicates that the criterion was not met.
ANALYSIS OF EQAO MATHEMATICS ASSESSMENTS
Mayes_04.20 EDUC783-Winter12
54
Table A1.17 Range-of-Knowledge Correspondence and Balance of Representation: Junior Assessment 2008
Strands and Sub-Strands Range of Objectives Range of Know.
Accept.c
Balance Indexd
Balance of Rep.
Accept.e Title Expec. # # Hita % of
Totalb
NSN1: Number Sense and Numeration 18 8 44 ❍ 0.90 ● NSN1: Quantity Relationships 7 4 57 ● 1 ●
NSN3: Operational Sense 8 2 25 ❍ 1 ● NSN4: Proportional Relationships 3 2 66 ● 0.83 ●
M: Measurement 12 7 58 ● 0.89 ● M1: Attributes, Units, and Meas. Sense 2 1 50 ● 1 ●
M2: Measurement Relationships 10 6 60 ● 0.88 ● GSS: Geometry and Spatial Sense 9 5 56 ● 0.77 ●
GSS1: Geometric Properties 4 2 50 ● 1 ● GSS2: Geometric Relationships 2 1 50 ● 1 ● GSS3: Location and Movement 3 2 66 ● 0.75 ●
PA: Patterning and Algebra 10 6 60 ● 0.77 ● PA1: Patterns and Relationships 6 4 66 ● 0.75 ●
PA2: Variables, Expressions, and Equations 4 2 50 ● 0.83 ● DMP: Data Management and Probability 12 6 50 ● 0.88 ● DMP1: Collection and Organization of Data 4 1 25 ❍ 1 ●
DMP2: Data Relationships 5 2 40 ❍ 0.83 ● DMP3: Probability 3 3 100 ● 1 ●
a # Hit: The number of expectations within a strand or a sub-strand that were hit at least once by items on the assessment. b % of Total: The percentage of expectations within a strand or a sub-strand that were hit by items on the assessment. c Range of Know. Accept.: ● indicates that 50% or more of the expectations within a strand or a sub-strand were hit to satisfy the Range-of-Knowledge criterion; ❍ indicates that this criterion was not met. d Balance Index: The balance index for the expectations hit within a strand or a sub-strand. e Balance of Rep. Accept.: ● indicates that the balance index for the strand or sub-strand was high enough to meet the Balance of Representation criterion (0.7 or above); ❍ indicates that the criterion was not met.
ANALYSIS OF EQAO MATHEMATICS ASSESSMENTS
Mayes_04.20 EDUC783-Winter12
55
Table A1.18
Range-of-Knowledge Correspondence and Balance of Representation: Junior Assessment 2009
Strands and Sub-Strands Range of Objectives Range of
Know. Accept.c
Balance Indexd
Balance of Rep.
Accept.e Title Expec. # # Hita % of
Totalb
NSN1: Number Sense and Numeration 18 7 39 ❍ 0.84 ● NSN1: Quantity Relationships 7 2 29 ❍ 0.83 ●
NSN3: Operational Sense 8 2 25 ❍ 0.83 ● NSN4: Proportional Relationships 3 3 100 ● 1 ●
M: Measurement 12 7 58 ● 0.81 ● M1: Attributes, Units, and Meas. Sense 2 0 0 ❍ N/A ●
M2: Measurement Relationships 10 7 70 ● 0.81 ● GSS: Geometry and Spatial Sense 9 5 56 ● 0.78 ●
GSS1: Geometric Properties 4 2 50 ● 1 ● GSS2: Geometric Relationships 2 0 0 ❍ N/A ● GSS3: Location and Movement 3 3 100 ● 0.73 ●
PA: Patterning and Algebra 10 5 50 ● 0.87 ● PA1: Patterns and Relationships 6 4 66 ● 1 ●
PA2: Variables, Expressions, and Equations 4 1 25 ❍ 1 ● DMP: Data Management and Probability 12 6 50 ● 0.83 ● DMP1: Collection and Organization of Data 4 2 50 ● 0.83 ●
DMP2: Data Relationships 5 1 20 ❍ 1 ● DMP3: Probability 3 3 100 ● 1 ●
a # Hit: The number of expectations within a strand or a sub-strand that were hit at least once by items on the assessment. b % of Total: The percentage of expectations within a strand or a sub-strand that were hit by items on the assessment. c Range of Know. Accept.: ● indicates that 50% or more of the expectations within a strand or a sub-strand were hit to satisfy the Range-of-Knowledge criterion; ❍ indicates that this criterion was not met. d Balance Index: The balance index for the expectations hit within a strand or a sub-strand. e Balance of Rep. Accept.: ● indicates that the balance index for the strand or sub-strand was high enough to meet the Balance of Representation criterion (0.7 or above); ❍ indicates that the criterion was not met.
ANALYSIS OF EQAO MATHEMATICS ASSESSMENTS
Mayes_04.20 EDUC783-Winter12
56
Table A1.19 Range-of-Knowledge Correspondence and Balance of Representation: Junior Assessment 2010
Strands and Sub-Strands Range of Objectives Range of
Know. Accept.c
Balance Indexd
Balance of Rep.
Accept.e Title Expec. # # Hita % of
Totalb
NSN1: Number Sense and Numeration 18 7 39 ❍ 0.84 ● NSN1: Quantity Relationships 7 2 29 ❍ 1 ●
NSN3: Operational Sense 8 2 25 ❍ 0.83 ● NSN4: Proportional Relationships 3 3 100 ● 0.83 ●
M: Measurement 12 6 50 ● 0.73 ● M1: Attributes, Units, and Meas. Sense 2 1 50 ● 1 ●
M2: Measurement Relationships 10 5 50 ● 0.76 ● GSS: Geometry and Spatial Sense 9 6 66 ● 0.88 ●
GSS1: Geometric Properties 4 2 50 ● 1 ● GSS2: Geometric Relationships 2 1 50 ● 1 ● GSS3: Location and Movement 3 3 100 ● 0.83 ●
PA: Patterning and Algebra 10 5 50 ● 0.83 ● PA1: Patterns and Relationships 6 4 67 ● 0.83 ●
PA2: Variables, Expressions, and Equations 4 1 25 ❍ 1 ● DMP: Data Management and Probability 12 3 25 ❍ 0.83 ● DMP1: Collection and Organization of Data 4 0 0 ❍ N/A ●
DMP2: Data Relationships 5 2 40 ❍ 0.75 ● DMP3: Probability 3 1 33 ❍ 1 ●
a # Hit: The number of expectations within a strand or a sub-strand that were hit at least once by items on the assessment. b % of Total: The percentage of expectations within a strand or a sub-strand that were hit by items on the assessment. c Range of Know. Accept.: ● indicates that 50% or more of the expectations within a strand or a sub-strand were hit to satisfy the Range-of-Knowledge criterion; ❍ indicates that this criterion was not met. d Balance Index: The balance index for the expectations hit within a strand or a sub-strand. e Balance of Rep. Accept.: ● indicates that the balance index for the strand or sub-strand was high enough to meet the Balance of Representation criterion (0.7 or above); ❍ indicates that the criterion was not met.
ANALYSIS OF EQAO MATHEMATICS ASSESSMENTS
Mayes_04.20 EDUC783-Winter12
57
Table A1.20 Range-of-Knowledge Correspondence and Balance of Representation: Junior Assessment 2011
Strands and Sub-Strands Range of Objectives Range of
Know. Accept.c
Balance Indexd
Balance of Rep.
Accept.e Title Expec. # # Hita % of
Totalb
NSN1: Number Sense and Numeration 18 7 39 ❍ 0.84 ● NSN1: Quantity Relationships 7 2 29 ❍ 1 ●
NSN3: Operational Sense 8 3 38 ❍ 0.87 ● NSN4: Proportional Relationships 3 2 66 ● 1 ●
M: Measurement 12 7 58 ● 0.89 ● M1: Attributes, Units, and Meas. Sense 2 1 50 ● 1 ●
M2: Measurement Relationships 10 6 60 ● 0.88 ● GSS: Geometry and Spatial Sense 9 5 56 ● 0.87 ●
GSS1: Geometric Properties 4 2 50 ● 1 ● GSS2: Geometric Relationships 2 1 50 ● 1 ● GSS3: Location and Movement 3 2 66 ● 0.83 ●
PA: Patterning and Algebra 10 8 80 ● 0.90 ● PA1: Patterns and Relationships 6 5 83 ● 0.87 ●
PA2: Variables, Expressions, and Equations 4 3 75 ● 1 ● DMP: Data Management and Probability 12 6 50 ● 0.78 ● DMP1: Collection and Organization of Data 4 2 50 ● 1 ●
DMP2: Data Relationships 5 2 40 ❍ 0.83 ● DMP3: Probability 3 2 66 ● 0.75 ●
a # Hit: The number of expectations within a strand or a sub-strand that were hit at least once by items on the assessment. b % of Total: The percentage of expectations within a strand or a sub-strand that were hit by items on the assessment. c Range of Know. Accept.: ● indicates that 50% or more of the expectations within a strand or a sub-strand were hit to satisfy the Range-of-Knowledge criterion; ❍ indicates that this criterion was not met. d Balance Index: The balance index for the expectations hit within a strand or a sub-strand. e Balance of Rep. Accept.: ● indicates that the balance index for the strand or sub-strand was high enough to meet the Balance of Representation criterion (0.7 or above); ❍ indicates that the criterion was not met.
ANALYSIS OF EQAO MATHEMATICS ASSESSMENTS
Mayes_04.20 EDUC783-Winter12
58
Appendix 2
Data comparing curricula defined by EQAO tests, The Ontario Curriculum, and
textbooks
Figure A2.1. Percentage of total hits on the 2007-2011 EQAO primary-level mathematics
assessments from each strand versus the percentage of grade 3 curriculum expectations
and textbook sections from each strand.
0
5
10
15
20
25
30
35
40
45
Number Sense
and Numeration
Measurement Patterning and
Algebra
Geometry and
Spatial Sense
Data Management
and Probability
Percent
EQAO Primary Assessments Curriculum Expectations Overall Textbook Content
ANALYSIS OF EQAO MATHEMATICS ASSESSMENTS
Mayes_04.20 EDUC783-Winter12
59
Figure A2.2. Percentage of total hits on the 2007-2011 EQAO junior-level mathematics
assessments from each strand versus the percentage of grade 6 curriculum expectations
and textbook sections from each strand.
0
5
10
15
20
25
30
35
40
45
50
Number Sense
and Numeration
Measurement Patterning and
Algebra
Geometry and
Spatial Sense
Data
Management and
Probability
Percent
EQAO Junior Assessments Curriculum Expectations Overall Textbook Content
ANALYSIS OF EQAO MATHEMATICS ASSESSMENTS
Mayes_04.20 EDUC783-Winter12
60
Table A2.1
Comparison between the content on the primary-level assessments by strands and the
content expected based upon the content of the curriculum
NSNa Mb GSSc PAd DMPe χ2 valuef
2007 +g 0h - 0 0 4.328 2008 0 0 0 0 0 1.939 2009 + 0 - 0 0 8.735 2010 + 0 0 0 0 2.307 2011 + 0 0 0 0 6.336 a NSN: Number Sense and Numeration strand. b M: Measurement strand c GSS: Geometry and Spatial Sense strand d PA: Patterning and Algebra strand DMP: Data Management and Probability strand e χ2 value: Gives the chi-square value (sum of the squared differences of expected value and actual value divided by the expected value) comparing the distribution of the content on each assessment to the expected distribution of content based on The Ontario Curriculum (number of expectations) or textbooks (number of sections). f +, -: Indicates that the difference between the actual number of hits and expected number of hits (based upon the curriculum or textbook content) is more than one standard deviation away from the mean of such differences. + signals that the assessment had more hits in a given strand than was expected. – signals that the assessment had fewer hits in a given strand than expected. g 0: Indicates that the difference between the actual number of hits and expected number of hits (based upon the curriculum or textbook content) is less than one standard deviation away from the mean of such differences.
ANALYSIS OF EQAO MATHEMATICS ASSESSMENTS
Mayes_04.20 EDUC783-Winter12
61
Table A2.2
Comparison between the content on the primary-level assessments by strands and the
content expected based upon the content of textbooks
NSNa Mb GSSc PAd DMPe χ2 valuef
2007 0g 0 0 0 0 0.885 2008 0 -f 0 0 0 1.581 2009 0 - 0 0 0 2.339 2010 0 0 0 0 0 1.733 2011 0 - 0 0 0 2.191 a NSN: Number Sense and Numeration strand. b M: Measurement strand c GSS: Geometry and Spatial Sense strand d PA: Patterning and Algebra strand DMP: Data Management and Probability strand e χ2 value: Gives the chi-square value (sum of the squared differences of expected value and actual value divided by the expected value) comparing the distribution of the content on each assessment to the expected distribution of content based on The Ontario Curriculum (number of expectations) or textbooks (number of sections). f +, -: Indicates that the difference between the actual number of hits and expected number of hits (based upon the curriculum or textbook content) is more than one standard deviation away from the mean of such differences. + signals that the assessment had more hits in a given strand than was expected. – signals that the assessment had fewer hits in a given strand than expected. g 0: Indicates that the difference between the actual number of hits and expected number of hits (based upon the curriculum or textbook content) is less than one standard deviation away from the mean of such differences.
ANALYSIS OF EQAO MATHEMATICS ASSESSMENTS
Mayes_04.20 EDUC783-Winter12
62
Table A2.3
Comparison between the content on the junior-level assessments by strands and the
content expected based upon the content of the curriculum
NSNa Mb GSSc PAd DMPe χ2 valuef
2007 -f 0 0 0 0 1.884 2008 0g 0 0 + 0 5.166 2009 0 0 0 0 + 2.381 2010 0 0 0 0 0 0.855 2011 0 0 0 0 + 4.636 a NSN: Number Sense and Numeration strand. b M: Measurement strand c GSS: Geometry and Spatial Sense strand d PA: Patterning and Algebra strand DMP: Data Management and Probability strand e χ2 value: Gives the chi-square value (sum of the squared differences of expected value and actual value divided by the expected value) comparing the distribution of the content on each assessment to the expected distribution of content based on The Ontario Curriculum (number of expectations) or textbooks (number of sections). f +, -: Indicates that the difference between the actual number of hits and expected number of hits (based upon the curriculum or textbook content) is more than one standard deviation away from the mean of such differences. + signals that the assessment had more hits in a given strand than was expected. – signals that the assessment had fewer hits in a given strand than expected. g 0: Indicates that the difference between the actual number of hits and expected number of hits (based upon the curriculum or textbook content) is less than one standard deviation away from the mean of such differences.
ANALYSIS OF EQAO MATHEMATICS ASSESSMENTS
Mayes_04.20 EDUC783-Winter12
63
Table A2.4
Comparison between the content on the junior-level assessments by strands and the
content expected based upon the content of textbooks
NSNa Mb GSSc PAd DMPe χ2 valuef
2007 -f 0g 0 0 0 18.638 2008 - 0 0 0 0 27.125 2009 - 0 0 0 0 14.806 2010 - 0 0 0 0 15.023 2011 - 0 0 0 0 18.416 a NSN: Number Sense and Numeration strand. b M: Measurement strand c GSS: Geometry and Spatial Sense strand d PA: Patterning and Algebra strand DMP: Data Management and Probability strand e χ2 value: Gives the chi-square value (sum of the squared differences of expected value and actual value divided by the expected value) comparing the distribution of the content on each assessment to the expected distribution of content based on The Ontario Curriculum (number of expectations) or textbooks (number of sections). f +, -: Indicates that the difference between the actual number of hits and expected number of hits (based upon the curriculum or textbook content) is more than one standard deviation away from the mean of such differences. + signals that the assessment had more hits in a given strand than was expected. – signals that the assessment had fewer hits in a given strand than expected. g 0: Indicates that the difference between the actual number of hits and expected number of hits (based upon the curriculum or textbook content) is less than one standard deviation away from the mean of such differences.
ANALYSIS OF EQAO MATHEMATICS ASSESSMENTS
Mayes_04.20 EDUC783-Winter12
64
Appendix 3
Notable Assessment Items
Figure A3.1: 2009 Primary Assessment, Question 29
Figure A3.2: 2007 Primary Assessment, Question 7
ANALYSIS OF EQAO MATHEMATICS ASSESSMENTS
Mayes_04.20 EDUC783-Winter12
65
Figure A3.3: 2008 Junior Assessment, Question 2
Figure A3.4: 2011 Primary Assessment, Question 9
ANALYSIS OF EQAO MATHEMATICS ASSESSMENTS
Mayes_04.20 EDUC783-Winter12
66
Figure A3.5: 2009 Primary Assessment, Question 7
ANALYSIS OF EQAO MATHEMATICS ASSESSMENTS
Mayes_04.20 EDUC783-Winter12
67
Figure A3.6: 2008 Junior Assessment, Question 28
Figure A3.7: 2011 Junior Assessment, Question 25
ANALYSIS OF EQAO MATHEMATICS ASSESSMENTS
Mayes_04.20 EDUC783-Winter12
68