understanding 2014–15 test scores act workkeys®, the act®, and act® aspire® presented by:...
TRANSCRIPT
Understanding 2014–15 Test Scores ACT WorkKeys®, The ACT®,
and ACT® Aspire®
Presented by:Kevin Fatica, Program Manager, ACT WorkKeys® & The ACT®
Sheila Graybeal, Program Manager, ACT® Aspire®
with assistance from Joe Saunders,Psychometric and Data Analysis Unit
Office of AssessmentSouth Carolina Department of Education
November 2015
This presentation refers to products and services of ACT, Inc. and ACT Aspire, LLC.The ACT® assessment and the ACT WorkKeys® assessment are products of ACT, Inc. The ACT® Aspire® assessments are products of ACT Aspire, LLC (founded in 2012 by partner companies ACT and Pearson). The ACT®, ACT WorkKeys®, and ACT® Aspire® are registered trademarks of ACT, Inc. The ACT National Career Readiness Certificate (NCRC) is a trademarked product of ACT, Inc. For more information about The ACT® and ACT WorkKeys®, visit http://act.org. For more information about ACT® Aspire®, visit http://discoveractaspire.org.
ACT WorkKeys®The ACT WorkKeys® assessment• A job skills assessment• Scores provide indications of job skills &
career readiness • Test items are based on workplace tasks &
materials• Students may be eligible for ACT’s National Career
Readiness Certificate
Source: http://act.org/products/workforce-act-workkeys/
ACT WorkKeys®
• The statewide testing program includesthree timed multiple-choice tests,taking 45 minutes each: • Reading for Information (33 items)• Applied Mathematics (33 items)• Locating Information (38 items)
ACT WorkKeys® ScoresRaw scores
• Number of items answered correctly
• No penalty for guessing
Scale Scores
• Transformations of raw scores
• Range from 65 to 90
Source: http://www.act.org/workkeys/assess/understanding.html?_ga=1.177381026.64531899.1446145893
ACT WorkKeys® Scores
• Level Scores
• Identify broad skill levels
• Levels range from 3–6 or 7 depending onthe test
• Scores can be used for hiring, promotion,or other individual purposes
ACT WorkKeys®Scale Score to Level Conversions
Reading for Information
Locating Information
Applied Mathematics
Level Scale Score Scale Score Scale Score<3 65-72 65-71 65-703 73-74 72-74 71-744 75-78 75-79 75-775 79-81 80-86 78-816 82-84 87-90 82-867 85-90 87-90
Source: http://www.act.org/workkeys/pdf/scale_score_interp_guide.pdf
ACT’s National Career Readiness Certificate (NCRC)
• Portable job skills credential
• Documents skills linked to workplace success
• Some employers have identified levels of ACT WorkKeys® scores as required or desirable for specific jobs
• Four certificate levels: Bronze, Silver, Gold,and Platinum
Source: http://act.org/certificate/index.html
ACT’s National Career Readiness Certificate (NCRC)
Bronze = scores at least a level 3 in each of the three core areas – indicates skills required for 16% of jobs in ACT’s job database
Silver = scores at least a level 4 in each of the three core areas– indicates skills required for 67% of jobs in ACT’s job database
Gold = scores at least a level 5 in each of the three core areas – indicates skills required for 93% of jobs in ACT’s job database
Platinum = scores at least a level 6 in each of the three core areas– indicates skills required for 99% of jobs in ACT’s job database
Notes: Percentages may change based on jobs in ACT’s database. Source: http://act.org/certificate/index.html
ACT WorkKeys® Reading for Information
• The score levels (3–7) build on each other. Each level incorporates skills assessed at the preceding level(s).
• Level 3 reading materials are short and direct.Items may ask about obvious details and provide short instructions.
• Level 6: complicated information, legal regulations, technical terms, elaborate procedures. Items require interpretation, application to new situation.
Source: http://www.act.org/workkeys/assess/reading/levels.html
ACT WorkKeys® Locating Information
• The score levels (3–6) build on each other. Each level incorporates skills assessed at the preceding level(s).
• Level 3 items may involve simple order forms, graphs, tables, charts, or other graphics. Examinees may find or fill in one or two pieces of information.
• As levels increase, the quantity, density, or complexity of the graphics and tasks increases.
• Level 6 items may involve complex graphics, large amounts of information, and may require examinees to draw conclusions, make decisions, or apply information.
Source: http://www.act.org/workkeys/assess/locate/levels.html
ACT WorkKeys® Applied Mathematics
• The score levels (3-7) build on each other. Each level incorporates skills assessed at the preceding level(s).
• Level 3 item characteristics: translate easily from a word problem to a math equation, require a single type of mathematical operation, may convert numbers from one form to another, or use money/time units.
• Level 7 item characteristics: multiple steps of logic and calculation, information may be incomplete or implicit, may require considerable translation from verbal to mathematical form, may apply basic statistical concepts, calculate volumes, or find the best deal from among several options.
Source: http://www.act.org/workkeys/assess/math/levels.html
ACT WorkKeys® Score Reports: Memo to Examinee
• Memo includes:• Level scores (per test)• Scale scores (per test) • Narrative explanation of skills expected of
person at skill level• Suggestions for improvement
ACT WorkKeys® Score Reports: Memo to Examinee
Source: http://www.act.org/workkeys/assess/reports.html
ACT WorkKeys® Score Reports: Summary for Examinee
• Summary includes:• Level scores (per test)• Scale scores (per test) • Narrative explanation of skills expected of
person at each skill level
ACT WorkKeys® Score Reports: Summary for Examinee
Source: http://www.act.org/workkeys/assess/reports.html
ACT WorkKeys® Results2015 South Carolina NCRC Levels
Additional Resources• Check with District Test Coordinators (DTCs) and School Test Coordinators (STCs)
• SCDE webpages• The ACT: https://ed.sc.gov/tests/high/the-act/ • ACT WorkKeys: https://ed.sc.gov/tests/high/act-workkeys/
• SCDE• Kevin Fatica, Program Manager
Office of AssessmentSouth Carolina Department of Education803-734-8282
• ACT, Inc.• ACT-hosted webpage (not public – see DTCs and STCs for address)• ACT, Inc. webpage http://act.org/ • The ACT and ACT WorkKeys General Inquiries
• Phone: (800) 553-6244, ext. 2800 • E-mail: [email protected]
Questions?
The ACT®
– Curriculum- and standards-based assessment– The ACT assesses students in: • English, Mathematics, Reading, and Science (multiple-choice items) • Writing (single writing prompt)
Source: http://act.org/products/k-12-act-test/
The ACT® Test in SC
English 75 MC* questions 45 minutes Measures standard written English and rhetorical
skills.
Mathematics 60 MC questions 60 minutes
Measures mathematical skills students have typically acquired in courses taken up to the beginning of grade 12.
Reading 40 MC questions 35 minutes Measures reading comprehension.
Science 40 MC questions 35 minutes
Measures the interpretation, analysis, evaluation, reasoning, and problem-solving skills required in the natural sciences.
Writing 1 prompt 40 minutes Measures writing skills emphasized in high school English classes and in entry-level college composition courses.
*The abbreviation MC refers to multiple choice items.
Source: http://act.org/products/k-12-act-test/
The ACT® Test Scores• English, Mathematics, Reading, and Science – Raw scores (number of correct responses) – Converted to scale scores– Range 1–36
• ACT Composite Score– Average of the four MC scale scores– Range 1–36
• Writing – Not part of composite score– Range 2–12
Source: http://act.org/products/k-12-act-test/
The ACT®: Subscores
• English– Usage and
Mechanics– Rhetorical Skills
• Mathematics– PreAlgebra/Algebra
– Algebra/Coord.
Geometry
Geometry/Trig
• Reading– Social Studies/
Sciences– Arts/Literature
Source: http://www.actstudent.org/scores/understand/studentreport14.html
The ACT®: Scoring of Writing• Scored independently by two readers using a 6-point holistic
scoring rubric • Writing score is the sum of the two ratings• Essays are evaluated on evidence that they: – demonstrate ability to make and articulate judgments– develop and sustain a position on an issue – organize and present ideas in a logical way – communicate clearly and effectively using conventions of
standard written English
Source: http://www.act.org/aap/pdf/ACT_Technical_Manual.pdf
The ACT®: College Readiness• Scores indicate readiness for college• Readiness is based on ACT’s College
Readiness Benchmark scores(except Writing)
• Benchmarks are per-subject area test• Students that meet or exceed the
Benchmark score are “Ready” • Those below the Benchmark are
“Not Ready”
The ACT® Benchmark ScoresACT Subject-Area Test The ACT® Benchmark College Course
English
18 English Composition
Mathematics
22 College Algebra
Reading
22 Social Sciences
Science
23 Biology
Source: http://www.act.org/solutions/college-career-readiness/college-readiness-benchmarks/
The ACT®: College Readiness
• A student scoring at the benchmark has approximately a 50% chance of receiving a grade of “B” or better and a 75% chance of receiving a “C” or better in the corresponding introductory college course.
• There is no benchmark for the Composite Score.• There is no benchmark score for Writing.
• Source: http://www.act.org/solutions/college-career-readiness/college-readiness-benchmarks/
Shows…• Composite Scores• Subject Area Scores• Subtest Scores• Ready/Not Ready• State and National
Percentile • Essay comments
The ACT® Student Score Reports
Source: http://www.actstudent.org/scores/understand/studentreport14.html
• Student score reports also contain additional information for guidance counselors, parents, or students
• May include colleges the student is interested in, career planning information, etc.
• Sample score report: http://www.actstudent.org/scores/understand/studentreport14.html
The ACT® Student Score Reports
The ACT® Student Score Reports
• If a student tested with State-Allowed Accommodations:– Scores were not college-reportable– ACT did not provide the standard score report– A score reporting letter was sent, providing all
scores in table format, along with guidance on score interpretation, and a note that the scores are not college-reportable
State-Allowed Accommodations Score Notification Letter
The ACT® 2015 South Carolina Mean Scale Scores
By Subject
* Writing scores are reported on a scale of 2 - 12
The ACT® 2015 South Carolina Percent Ready
Additional Resources• Check with District Test Coordinators (DTCs) and School Test Coordinators (STCs)
• SCDE webpages• The ACT: https://ed.sc.gov/tests/high/the-act/ • ACT WorkKeys: https://ed.sc.gov/tests/high/act-workkeys/
• SCDE• Kevin Fatica, Program Manager
Office of AssessmentSouth Carolina Department of Education803-734-8282
• ACT, Inc.• ACT-hosted webpage (not public – see DTCs and STCs for address)• ACT, Inc. webpage http://act.org/ • The ACT and ACT WorkKeys General Inquiries
• Phone: (800) 553-6244, ext. 2800 • E-mail: [email protected]
Questions?
ACT® Aspire® Overview• ACT Aspire includes vertically-scaled achievement tests designed
to measure student growth in a longitudinal assessment system forgrades 3–10 in English, reading, writing, mathematics, and science.
• ACT Aspire is designed to measure students’ progress toward collegeand career readiness. For all subjects except writing, the scale scoresare linked to college and career data through scores for The ACT® andthe ACT National Career Readiness Certificate (ACT NCRC®) program.
• The content of the grade 8 ACT Aspire tests is based on the ACT College and Career Readiness Standards (as identified through research including the ACT National Curriculum Survey). For grades 3–7, the ACT Aspire tests are reportedly consistent with many state’s standards that are focused on college and career readiness (such as the Common Core State Standards for ELA and mathematics).
Source: ACT® Aspire® Summative Assessment Technical Bulletin #1
ACT® Aspire® Administration in SC, Spring 2015
Subject Grades Time ItemTypes*
TotalItems
OperationalItems
[MC, CR]
OperationalPoints
[MC, CR]
Writing 3–56–8 30 min. prompt 1 1 20
24
English 3–56–8
30 min.35 min. MC 32
452535
2535
Reading 3–78 60 min. MC, CR 31 24 [21, 3] 29 [21, 8]
31 [21, 10]
Math3–5
6 & 78
55 min.60 min.65 min.
MC, CR314145
25 [21, 4]34 [30, 4]38 [33, 5]
37 [21, 16]46 [30, 16]53 [33, 20]
Note: Districts also had an option of administering ACT Aspire science in any of grades 3–8, a 55-minute test, including MC and CR items at all grades levels.
*MC = multiple-choice, CR = constructed-response
ACT® Aspire®: Types of Scores• Raw scores – total points earned on all questions• Scale scores – transformed version of raw scores
– Scales scores are vertically calibrated in every subject except writing.
– Scales scores are 3-digit numbers with a minimum of 400 (408 for writing) for all grades and a maximum that varies by grade and subject but is never greater than 460.
• Readiness levels – four categories of achievement1. In Need of Support2. Close3. Ready4. Exceeding (reported only on the data file)
ACT® Aspire®: Determining Readiness• ACT Aspire readiness levels are designed to be predictive of
ACT Readiness Benchmarks (for all subjects except writing).• ACT Aspire tests in grades 8–10 have been equated
to existing EXPLORE and PLAN tests, which have empirical data for predicting results on The ACT.
• ACT Aspire tests in grades 3–7 have readiness criteria that have been extrapolated from grade 8 using theZ-score method. – There is, as yet, no empirical data to validate those predictions
directly. (The 1st operational administrationof ACT Aspire in any school district or state occurred inthe 2013–14 school year.)
ACT® Aspire®: Defining the Levels• After the readiness criteria were determined for each grade and
subject, an approximate 95% confidence interval was determined for each readiness score.
• The lower and upper limits of the confidence interval (the scores that fell ± 2 standard errors of measurement from the readiness score) defined the criteria separating levels 1 and 2 (In Need of Support and Close) and levels 3 and 4 (Ready and Exceeding).
• Levels are based on the probability of being “Ready,” rather than on specific content and abilities.
• Levels were predetermined by the test vendor, rather than by committees of South Carolina educators.
Source: ACT® Aspire® Summative Assessment Technical Bulletin #2
ACT® Aspire®: Scoring of Writing• The ACT Aspire Writing scale scores are rubric-driven and based on
four domains (Ideas and Analysis, Development and Support, Organization, and Language Use and Conventions), with each domain scored on a scale of 1–5 points (grades 3–5) or 1–6 points (grades 6–8). (The rubrics become more complex across grades.)
• The total raw writing score is the sum of the four raw domain scores and ranges from 4 to 20 for grades 3–5 and from 4 to 24 for grades 6–8.
• For all grades, the “Ready” cut-score was set as the scale score that corresponds to two 4s and two 3s on the four domains. A more detailed description of the scoring and scaling of Writing can be found in the ACT Aspire Summative Assessment Technical Bulletin #2 (pages 79– 80).
• A common linear function shared by all grades was used to convert the total raw scores to three-digit ACT Aspire scale scores for the writing forms. The lowest obtainable scale score for writing is 408 for all grades, and the highest obtainable scale score is 440 for grades 3–5 and 448 for grades 6–8.
Source: ACT® Aspire® Summative Assessment Technical Bulletin #2
ACT® Aspire®: Interpreting Scale Scores• Scale scores for English, reading, mathematics, and science are vertically
scaled.– This implies that a score of 423 in English, for example, reflects the same level of
achievement at any grade level– It also implies that a student’s scale scores in a subject would be expected increase
across grade levels if the student were progressing• The scoring process for writing implies that the writing scale scores do not
have a vertical scale interpretation like those obtained for the other subjects. – Writing scale scores across grade levels reflect performance on the rubric-driven
expectations; the underlying performance needed to receive the same score is automatically adjusted to grade-level expectations.
– Consistency in scores across grade levels suggests the same performance relative to grade-level expectations. For example, if a student receives a score of 432 on the Writing test at grade 3 and a score of 432 at grade 8,it means that the student has achieved a consistent level of performance relative to grade-level expectations. In other words, the same writingscale scores across grades 3 and 8 would that a student had grown to meet grade-level expectations.
ACT® Aspire®: More about Writing Scoring
• The ACT Aspire writing responses were scored by human readers, with the majority (about 70%) scored by only one reader.
• During the scoring process, a response may have been determined to be “non-scorable.”
• Explanations for non-scorable responses included the following:– blank answer document,– “invalidate/void” gridded on the answer document (INV),– not English (NE),– illegible (IL), and– off-topic (OT).
• Codes for these non-scoreable responses (INV, NE, etc.) appearedin the Writing scale score column in the updated data files (posted to the ACT Aspire portal on August 5).
• Reasons for particular codes such as OT may have been due to varied circumstances (e.g., copied prompt, miscoded form).
ACT® Aspire®: Normative Data• The following tables display normative data associated with the ACT Aspire cut-points for
each subject test and grade level.• The norms are defined as “the cumulative percent of students scoring at or below a
given score in the norm sample,” and are listed in the columns headed “% AB.”• While the ACT Aspire norms may appear similar to the national percentile ranks that are
generally reported for norm-referenced tests, they are not identical.The major difference is that the ACT Aspire norms are based on a user sample (students tested during a 2013 special study and operationally in 2014) rather than a nationally representative sample (students selected to represent the national student population in terms of demographic characteristics).
• Characteristics of the ACT Aspire norm group are presented in Chapter 3 of the ACT Aspire Summative Assessment Technical Bulletin #2.
• As indicated in the following tables, there are some obvious differences in the% AB values that correspond to the ACT Aspire “Ready” (and other) cut-scores across and within subjects.
• Across subjects, for example, English appears to be the least rigorous test andwriting the most rigorous. Depending on the grade level, a student who meetsthe minimum criterion for Ready on the English test scored as well or better than 31 to 38 percent of students in the norm group (depending on the grade level), while a student at the Ready cut-score for writing scored as well or better than61 to 89 percent of students in the norm group.
ACT® Aspire® Cut-Scores and % At or BelowWriting and English
SubjectGrade
Tested
Close (Low
Cut)
Close % AB
Ready (Benchmark)
Ready% AB
Exceeding(High Cut)
Exceeding% AB
Writing 3 420 51 428 89 436 99 4 420 40 428 87 436 99 5 420 35 428 78 436 97 6 420 22 428 61 436 92 7 420 40 428 81 436 98 8 420 39 428 81 436 98
English 3 408 8 413 36 418 66 4 411 9 417 38 423 72 5 412 7 419 37 426 72 6 413 10 420 35 427 68 7 413 10 421 33 429 67 8 415 10 422 31 429 61
ACT® Aspire® Cut-Scores and % At or BelowReading and Math
SubjectGradeTested
Close(Low Cut)
Close % AB
Ready(Benchmark)
Ready% AB
Exceeding(High Cut)
Exceeding% AB
Reading 3 411 48 415 72 419 91 4 412 36 417 68 422 90 5 415 40 420 73 425 91 6 416 35 421 61 426 87 7 417 38 423 71 429 96 8 418 32 424 58 430 89
Math 3 409 22 413 58 417 89 4 411 17 416 65 421 93 5 412 16 418 64 424 91 6 414 22 420 63 426 89 7 416 39 422 71 428 91 8 419 46 425 72 431 88
ACT® Aspire®: Reporting for Skill Areas
• In addition to ACT Aspire scale scores, readiness levels, and percentile ranks for the overall subject test, information on student performance in skill areas is reported for each subject. Reporting categories may vary by grade level within a subject. Examples are provided below.
• Writing: Ideas and Analysis, Development and Support, Organization, Language Use and Conventions
• English: Production of Writing, Knowledge of Language (gr 4–8 only), Conventions of Standard English
• Reading: Key Ideas and Details, Craft and Structure, Integration of Knowledge and Ideas• Math: Foundation, Justification and Explanation, Modeling, Number and Operations–
Fractions (gr 3–5), Number & Operations in Base 10 (gr 3–5), The Number System (gr 6–8), Operations & Algebraic Thinking (gr 3–5), Expressions & Equations (gr 6–8), Ratios & Proportional Relationships(gr 6, 7), Functions (gr 8), Geometry, Measurement & Data (gr 3–5), Statistics & Probability (gr 6–8)
See Appendix A of the Interpretive Guide for ACT Aspire Summative Reports, on the ACT Aspire website.
ACT® Aspire®:Skill Areas & Supplemental Scores
• For each skill area reporting category, the percent and number of points earned out of the possible number of points is reported.
• Student performance in each category is also compared toscores of students who meet the ACT Readiness Benchmark (indicated as the “ACT Readiness Range” for the category).
• Supplemental scores that may be reported include:– ELA (average of writing, English, and reading scores; ready or below);– Progress with Text Complexity (sufficient progress: yes or no)– STEM (average of mathematics and optional science scores; ready or
below); and,– at grade 8 only, a composite score (average of English, reading,
mathematics, and optional science scores) and corresponding progress toward Career Readiness indicator.
ACT® Aspire®: Reliability & Cautions• As with any statewide summative test results, district and school users
should keep in mind the limitations of the ACT Aspire results.• Test data constitute a single source of information that should be used in
conjunction with other relevant information when evaluating educational quality or making instructional decisions.
• As noted in the Interpretive Guide for ACT Aspire Summative Reports(page 1) as well as on the individual student reports, “(s)cores are estimates of student knowledge and achievement. The student’s actual achievement could be slightly higher or lower than the reported scores.” (See chapter 8 of the ACT Aspire Summative Assessment Technical Bulletin #2 for detailed information about the reliability of ACT Aspire tests.)
• Concerns about reliability and measurement error are magnified when considering student-level results for the reporting categories or skill areas. In some cases, these results are based on an extremely small number of items or points (e.g., as small as two or three).
Spring 2015 ACT® Aspire®Data Files & Reports
The following spring 2015 data files and score reports were provided to districts for ACT Aspire: • district-level Student Performance Files (SPFs), posted to
the ACT Aspire portal on July 11,• Individual Student Reports (ISRs), posted to the portal July 17, • District/School Summary Reports, posted to the portal July 24, • paper ISRs (1 per student), boxed by school, shipped to DTCs for
arrival by July 29, and• updated district-level SPFs posted to the portal August 5.
Note: Access to the ACT Aspire portal ended on November 1, 2015. Data and reports should have been retrieved and downloaded prior to that date.
ACT® Aspire® Score ReportsIndividual Student Report, p. 1
ACT® Aspire® Score ReportsIndividual Student Report, p. 2
ACT® Aspire® Score ReportsIndividual Student Report, p. 3
ACT® Aspire® Score ReportsCurrent Progress (School, District)
ACT® Aspire® Score ReportsSupplemental Scores (School, District)
ACT® Aspire® Score ReportsSubject Proficiency by Demographic (School, District)
ACT® Aspire® Score ReportsSubject Proficiency by Grade Level (School, District)
ACT® Aspire® Score ReportsSubject Proficiency by School (District-Level)
ACT® Aspire® Results for SC, Spring 2015
ACT® Aspire® Results for SC, Spring 2015
3 4 5 6 7 8410.0
415.0
420.0
425.0
430.0
Mean Scale Scores by Subject
EnglishReadingWritingMathematics
Grade
Mea
n Sc
ale
Scor
e
Sub-ject
ACT® Aspire ® Additional Resources
• Check with District Test Coordinators (DTCs) and School Test Coordinators (STCs)• SCDE ACT Aspire Webpage: http://www.ed.sc.gov/tests/elementary/act-aspire/• SCDE:
Sheila Graybeal, Program ManagerOffice of AssessmentSouth Carolina Department of Education803-734-8649, [email protected]
• ACT Aspire:– ACT Aspire Webpage: http://discoveractaspire.org/assessments/summative/
(scroll down for Technical Bulletin #1 and Technical Bulletin #2)– ACT Aspire Launch Page: http://actaspire.pearson.com
(click on “Reporting” tab, then “Summative Reporting Resources”) – ACT Aspire Customer Support (General Inquiries)
• Phone: (888) 802-7502 • E-mail: [email protected], or
reporting-related inquiries, [email protected]
Questions?
Thank You!