virginia modified achievement standards test (vmast
TRANSCRIPT
![Page 1: Virginia Modified Achievement Standards Test (VMAST](https://reader033.vdocument.in/reader033/viewer/2022051412/627dec5bba86fe5871492de4/html5/thumbnails/1.jpg)
2011 National Conference on Student Assessment Orlando, Florida
Virginia Modified Achievement Standards Test (VMAST):
Lessons Learned
Shelley Loving-Ryder Virginia Department of Education
June 20, 2011
![Page 2: Virginia Modified Achievement Standards Test (VMAST](https://reader033.vdocument.in/reader033/viewer/2022051412/627dec5bba86fe5871492de4/html5/thumbnails/2.jpg)
2011 National Conference on Student Assessment Orlando, Florida
VMAST: Overview
• GSEG to develop assessment for Grade 8 reading and mathematics
• Adds supports and simplifications recommended by educators to existing online test items
![Page 3: Virginia Modified Achievement Standards Test (VMAST](https://reader033.vdocument.in/reader033/viewer/2022051412/627dec5bba86fe5871492de4/html5/thumbnails/3.jpg)
2011 National Conference on Student Assessment Orlando, Florida
• Conducted focus groups and online surveys with teachers and special education directors on learning characteristics and support/simplifications used with identified students in instruction
• In fall 2008 and spring 2009 conducted 2 pilots using cognitive interviews with students from urban and rural school districts who were identified as likely VMAST participants
- 3 -
Activities to Date
![Page 4: Virginia Modified Achievement Standards Test (VMAST](https://reader033.vdocument.in/reader033/viewer/2022051412/627dec5bba86fe5871492de4/html5/thumbnails/4.jpg)
2011 National Conference on Student Assessment Orlando, Florida
Field Test • Field test with approximately 500 students in
May-June, 2010 for VMAST grade 8 reading and mathematics.
• Students participating in field test were students with disabilities who met the participation criteria for VMAST
• 47 field test students were administered unsupported items first and were then administered the supported items
![Page 5: Virginia Modified Achievement Standards Test (VMAST](https://reader033.vdocument.in/reader033/viewer/2022051412/627dec5bba86fe5871492de4/html5/thumbnails/5.jpg)
2011 National Conference on Student Assessment Orlando, Florida
Results: Reliability
Subject N-Count Alpha VMAST
Alpha SOL
Math 235 0.85 0.89 - 0.93
Reading 257 0.84 0.88 – 0.90
![Page 6: Virginia Modified Achievement Standards Test (VMAST](https://reader033.vdocument.in/reader033/viewer/2022051412/627dec5bba86fe5871492de4/html5/thumbnails/6.jpg)
2011 National Conference on Student Assessment Orlando, Florida
Results of the Mathematics Field Test
![Page 7: Virginia Modified Achievement Standards Test (VMAST](https://reader033.vdocument.in/reader033/viewer/2022051412/627dec5bba86fe5871492de4/html5/thumbnails/7.jpg)
2011 National Conference on Student Assessment Orlando, Florida
Math Supports and Simplifications • Provided hint boxes with formulas,
strategies, and reminders, • Color coded important information • Provided online manipulatives • Simplified numbers • Simplified language in stems • Added or altered graphics • Reduced answer options from 4 to 3
![Page 8: Virginia Modified Achievement Standards Test (VMAST](https://reader033.vdocument.in/reader033/viewer/2022051412/627dec5bba86fe5871492de4/html5/thumbnails/8.jpg)
2011 National Conference on Student Assessment Orlando, Florida
Overall Comparison of VMAST vs SOL for Grade 8 Mathematics
VMAST is 40 MC items (vs 50 on the SOL test) measuring: • RC1: Number and Number Sense; • RC2: Computation and Estimation; • RC3: Measurement and Geometry; • RC4: Probability and Statistics; and • RC5: Patterns, Functions and Algebra.
Test N Average Percent Correct
Std Dev Min Max
VMAST 47 59 17.52 8 95 SOL 47 48 15.28 20 80
![Page 9: Virginia Modified Achievement Standards Test (VMAST](https://reader033.vdocument.in/reader033/viewer/2022051412/627dec5bba86fe5871492de4/html5/thumbnails/9.jpg)
2011 National Conference on Student Assessment Orlando, Florida
Mathematics Item-Level Analyses
• Change in p-values for supported items • 5 items had an increase in p-value
from .20 to .34 • 21 items had an increase in p-value
from.10 to .19 • 8 items either had no change in p-value
or the increase was less than .10 • 6 items had a decrease in p-value
![Page 10: Virginia Modified Achievement Standards Test (VMAST](https://reader033.vdocument.in/reader033/viewer/2022051412/627dec5bba86fe5871492de4/html5/thumbnails/10.jpg)
2011 National Conference on Student Assessment Orlando, Florida
Item-Level Analyses: Overall Results
• P-Value • Ranged from
.34 to .90 • Mean of .65
• Point biserial • Ranged from
.11 to .59
![Page 11: Virginia Modified Achievement Standards Test (VMAST](https://reader033.vdocument.in/reader033/viewer/2022051412/627dec5bba86fe5871492de4/html5/thumbnails/11.jpg)
2011 National Conference on Student Assessment Orlando, Florida
Recap of Mathematics Results • Reliability and Validity look good • Students have higher percent correct on VMAST
than on SOL • No atypical patterns in gender/ethnicity results • Nice range in item difficulty and item
discrimination • Some supported items became more difficult
• Consider number of supports added • Were supports used by students?
![Page 12: Virginia Modified Achievement Standards Test (VMAST](https://reader033.vdocument.in/reader033/viewer/2022051412/627dec5bba86fe5871492de4/html5/thumbnails/12.jpg)
2011 National Conference on Student Assessment Orlando, Florida
Results of the Reading Field Test
![Page 13: Virginia Modified Achievement Standards Test (VMAST](https://reader033.vdocument.in/reader033/viewer/2022051412/627dec5bba86fe5871492de4/html5/thumbnails/13.jpg)
2011 National Conference on Student Assessment Orlando, Florida
Reading Supports and Simplifications • Shortened reading passages • Excerpted relevant sections of reading passages
and presented them with item • Provided hint boxes with reminder to return to
passage when item dealt with passage as whole • Used graphic organizers to present passage
information • Simplified language in stems • Highlighted key words • Reduced answer options from 4 to 3
![Page 14: Virginia Modified Achievement Standards Test (VMAST](https://reader033.vdocument.in/reader033/viewer/2022051412/627dec5bba86fe5871492de4/html5/thumbnails/14.jpg)
2011 National Conference on Student Assessment Orlando, Florida
Overall Comparison of VMAST vs SOL for Grade 8 Reading
VMAST is 36 MC items (vs 45 on the SOL test) measuring:
• RC1: Use word analysis strategies and information resources; and • RC2: Demonstrate comprehension of printed materials.
Test N Average Percent Correct
Std Dev Min Max
VMAST 47 60 17.93 22 94
SOL 47 44 16.76 11 84
![Page 15: Virginia Modified Achievement Standards Test (VMAST](https://reader033.vdocument.in/reader033/viewer/2022051412/627dec5bba86fe5871492de4/html5/thumbnails/15.jpg)
2011 National Conference on Student Assessment Orlando, Florida
Item-Level Analyses: Overall Results
• P-Value • Ranged from
.29 to .90 • Mean of .62
• Point Biserial • Ranged from .22 to .57
![Page 16: Virginia Modified Achievement Standards Test (VMAST](https://reader033.vdocument.in/reader033/viewer/2022051412/627dec5bba86fe5871492de4/html5/thumbnails/16.jpg)
2011 National Conference on Student Assessment Orlando, Florida
Reading Item-Level Analyses
• Impact of supports and simplifications on reading VMAST items – 10 items had an increase in p-value
from .20-.38 – 16 items had an increase in p-value
from .10-.19 – 10 items had p-values that were either
unchanged or had an increase of less than .10
![Page 17: Virginia Modified Achievement Standards Test (VMAST](https://reader033.vdocument.in/reader033/viewer/2022051412/627dec5bba86fe5871492de4/html5/thumbnails/17.jpg)
2011 National Conference on Student Assessment Orlando, Florida
Recap of Reading Results
• Reliability and Validity look good • Students have higher percent correct on
VMAST than on SOL • No atypical patterns in gender/ethnicity results • Nice range in item difficulty and item
discrimination • Passage type may play a role in effectiveness
of the supports and simplifications
![Page 18: Virginia Modified Achievement Standards Test (VMAST](https://reader033.vdocument.in/reader033/viewer/2022051412/627dec5bba86fe5871492de4/html5/thumbnails/18.jpg)
2011 National Conference on Student Assessment Orlando, Florida
Lessons Learned: Effective Supports
• Hint boxes • Simplifying language • Simplifying presentation of item • Drag and drop tool to fill in equations or
expressions • Color coding when used judiciously • Highlighting • Excerpted text • Reminder to return to passage when item
addresses passage as a whole
![Page 19: Virginia Modified Achievement Standards Test (VMAST](https://reader033.vdocument.in/reader033/viewer/2022051412/627dec5bba86fe5871492de4/html5/thumbnails/19.jpg)
2011 National Conference on Student Assessment Orlando, Florida
Lessons Learned Supports are most helpful when they are consistently displayed across items • Hint boxes always in the same location • Components that can be dragged and
dropped identified consistently • Same “icon” to activate animation
- 19 -
![Page 20: Virginia Modified Achievement Standards Test (VMAST](https://reader033.vdocument.in/reader033/viewer/2022051412/627dec5bba86fe5871492de4/html5/thumbnails/20.jpg)
2011 National Conference on Student Assessment Orlando, Florida
Lessons Learned • Must be judicious in use of supports:
“oversupporting” can be confusing to students
• Supports and simplifications not helpful if students don’t know content
• Providing practice with items is crucial. Practice items and guides for teachers may be found at www.doe.virginia.gov/testing/alternative_assessments/vmast_va_mod_achievement_stds_test/practice_items/index.shtml
![Page 21: Virginia Modified Achievement Standards Test (VMAST](https://reader033.vdocument.in/reader033/viewer/2022051412/627dec5bba86fe5871492de4/html5/thumbnails/21.jpg)
2011 National Conference on Student Assessment Orlando, Florida
Next Steps • Moving toward implementation of VMAST in
reading and mathematics in grades 3-8, Algebra I and end-of-course reading
• Field tested mathematics in spring 2011 • First operational administration of
mathematics in spring 2012 • Field test reading in spring 2012 • First operational administration of reading in
spring 2013
![Page 22: Virginia Modified Achievement Standards Test (VMAST](https://reader033.vdocument.in/reader033/viewer/2022051412/627dec5bba86fe5871492de4/html5/thumbnails/22.jpg)
2011 National Conference on Student Assessment Orlando, Florida
Application to Next Generation Assessments
• Continue to study the characteristics of students to ensure that participation criteria are clear and appropriate
• Use principles of universal design in item development • Reduce language load • Simplify presentation of items
• Maximize the use of technology to develop additional supports and simplifications
![Page 23: Virginia Modified Achievement Standards Test (VMAST](https://reader033.vdocument.in/reader033/viewer/2022051412/627dec5bba86fe5871492de4/html5/thumbnails/23.jpg)
2011 National Conference on Student Assessment Orlando, Florida
Contact Information:
Shelley Loving-Ryder Virginia Department of Education
[email protected] (804) 225-2102