balancing test security and accessibility on next generation online assessments

National Center on Educational Outcomes (NCEO)

Balancing Test Security and Accessibility on Next Generation

Online Assessments

Martha Thurlow Sheryl Lazarus NCEO NCEO

John Fremer Shelley Loving-Ryder Caveon Virginia Department of Education

National Conference on Student AssessmentNew Orleans, Louisiana

June 25, 2014, 2013

Overview

I. The issues

II. Standards and best practices documents that address cheating or test security and accessibility

III. States’ test security policies

IV. Test security company perspective

V. State perspective

VI. Summary and discussion

VII.Questions


I. The Issues


• Test security policies and procedures need to address accommodations and other issues related to students with disabilities to help ensure the integrity and validity of a test.

• For example, some accommodations that individuals with disabilities and English language learners may need to meaningfully access a test:

- Require that someone other than the student sees the test.

- Require the use of specialized accessibility software.

- Require individual or small group individual administration.

• Alternate assessments may have some unique test security issues.

Example: Accessibility Software


Issue—

Some students with disabilities use accessibility software during instruction.

Computer-based tests sometimes require that the accessibility software be disabled (though there may be test-based accessibility features).

Result—

Student is denied meaningful access to assessment.

II. Standards and Guidelines that Address Cheating or Test Security

• Standards for Educational and Psychology Testing (APA/AERA/NCME, 1999)—Revised version will be available soon.

• Operational Best Practices for Statewide Large-Scale Assessment Programs (ATP/CCSSO, 2013)

• Testing and Data Integrity in the Assessment of Student Achievement (NCME, 2012)

• Other relevant documents


Standards for Educational and Psychology Testing

The Standards indicate that:

“all examinees be given a comparable opportunity to demonstrate their standing on the construct(s) the test is intended to measure. Just treatment also includes such factors as the appropriate testing conditions and equal opportunity to become familiar with the test format, practice materials, and so forth. Fairness also requires that all examinees be afforded appropriate testing conditions.” (p. 74)


Standards for Educational and Psychology Testing

The Standards indicate that:

“test users have the responsibility of protecting the security of test materials at all times.”


Operational Best Practices for Statewide Large-Scale Assessment Programs


The Operational Best Practices indicate that the document addresses:

“best practices for achieving test security throughout the process of all paper-based and technology-based assessments, using methods that support the needs of students with disabilities and English learners.” (p. 81)

Operational Best Practices for Statewide Large-Scale Assessment Programs,

Cont.


Chapter 19. Assessment of Special Populations/ Accessibility for All Learners

•Universal Design•Accommodations•Alternate Assessments

Testing and Data Integrity in the Administration of Statewide Student

Assessment Programs (NCME)


Policies and procedures must ensure that all students have appropriate, fair, and equal opportunities to show their knowledge, skills, and abilities. Students who need accommodations due to language differences or students with disabilities may require appropriate modifications to materials and administrative procedures to ensure fair access to the assessment of their skills.

Other Relevant Documents


• TILSA Test Security Guidebook (Olson & Fremer, 2013)

• Testing Integrity Symposium: Issues and Recommendations for Best Practice (NCES, 2013)

.

.

Analysis of States’ Test Security Policies

Source: Test security and students with disabilities: An analysis of states’ 2013-14 test security policies (Synthesis Report 95). National Center on Educational Outcomes. Available at: http://www.cehd.umn.edu/nceo/OnlinePubs/Synthesis95/SynthesisReport95.pdf

• What information do states’ test security policies contain about students with disabilities and the assessments they take?

• How are accommodations included in states’ test security policies?


Research Questions

• State policies downloaded for the 50 states + DC

• Documents analyzed included: Test procedures or administration manuals, general test security documents, accommodations manuals.

• Analyzed documents for both the regular assessment and the alternate assessment.


Procedures

Number of States’ Test Security Policies That Addressed Selected Issues

.

Duplication and Copying of Test Materials

.

Allowed

‘School staff members may reproduce secure test materials, in whole or in part, if this is necessary to provide a particular test accommodation for a student with an Individualized Education Program (IEP) or a Section 504 Plan.”

Not Allowed

“It is a violation of [test] security to: Copy, reproduce, or use in any manner any portion of any secure assessment book, for any reason. Student is allowed to use assistive technology to magnify/enlarge. The actual test booklet may not be copied or enlarged.”

State Examples

Duplication and Copying of Test Materials, cont.


“One week prior to the opening of the test administration window, the materials may be printed and then photocopied from a master copy for individual student use.”

State Example: Alternate Assessment Based on Alternate Achievement Standards (AA-AAS)

Role of IEP


“Teachers and other staff members may give accommodations or modifications that are consistent with the student’s IEP, 504, or LEP Plan and that are used in the normal delivery of instruction with the exception of those indicated as not allowable in the Accommodations Guidelines.”

State Example

Training Requirements


• “Ensure all personnel involved with testing receive training on appropriate test administration, policies, and procedures including accommodations.”

• “Assessment proctors who are in ‘accommodated sessions’ should receive training on the appropriate way to administer that specific assessment accommodation. Test proctors who administer the assessment to students with multiple accommodations must be trained on all applicable accommodations.”

State Examples

Qualifications


• “Test administrators shall be licensed personnel.”

• “Any person providing an accommodation must be an adult non-relative of the student. Students may not serve as scribes for other students, even if they are older students.”

State Examples

Security of Test Materials


“Secure test materials include: all customized administration materials including Braille, sign language, large-print, and loose-leaf test booklets, oral administration scripts (OASs), audio CD-ROMS, signed administration scripts (SASs), sign language DVDs, Braille Test Administrator’s Notes (TA Notes), and Braille OASs. Secure materials must be kept in secure, locked storage when not in use.”

State Example

Test Security & Alternate Assessments

27 States have test security policies that specifically addressed alternate assessments.

Alternate Assessments, cont.


State Examples

Item-based performance tasks:

“Test Books and student Answer Documents must be kept secure. Students must not be exposed to test content before the actual testing. If students have prior knowledge of test content, results of testing can give a deceptive picture.”

Performance tasks: “Tasks are secure materials and should not be sent home with the student.”

Alternate Assessment Test Security, cont.


State Examples

Portfolios:

“Once the portfolios contain student identifying information, student testing materials, and student work, the portfolios become secure documents and must be treated with the same care as other secure testing materials, keeping them in a locked file cabinet.”

Checklist:

Checklist is not a secure document. This means that the checklist may be copied.

Accommodations Most Frequently Addressed in States’ Test Security

Policies

.

Braille

.

State Example

“Under the supervision of the District or School Assessment Coordinator, a Test Monitor administering a Braille test may access the Test Administrator Notes and the corresponding test books up to 48 hours prior to the scheduled administration.”


Sign Interpretation


State Examples

• ‘Sign language presentation may be provided for all directions and test items and answer choices on mathematics and science tests. Test items may not be reworded, summarized, or simplified. When providing answer choices, the test administrator must be careful not to use language that might lead a student to the correct/incorrect responses.”

• “A student’s teacher should not serve as the interpreter/translator in a testing situation unless a second person is present to monitor for quality and fairness.”

Oral Delivery Accommodations

.

Human Reader


• “Test administrators conducting an oral administration must be aware that they are viewing secure content and that responding to test questions, recording the information they see, scoring the test, or discussing the content of the test at any time is strictly prohibited. As a reminder of this obligation, individuals who give an oral administration of a test are required to sign a separate section of the test administrator’s oath.”

• “Test administrator must read in a pace and tone that is appropriate for each individual student. Careful attention must be given such that no changes in tone or inflection are detectable which might indicate a correct answer.”

State Examples

Text to Speech


“If a text-to-speech program is being utilized, I have reviewed the material to ensure the reading passages will not be read aloud, and the content of the test is pronounced appropriately and the test items are not read in a manner to give away an answer.”.

State Example

Scheduling/Timing Accommodations

.

Setting Accommodations

.

State Approaches to Test Security


Good Practice

•Test security guidelines that address accommodations and alternate assessments.

•Test security agreements address accommodations.

See questions in report for list of things states and consortia may want to consider when balancing test security and accessibility.


For More Information

National Center on Educational Outcomeswww.nceo.info

Sheryl [email protected]

Martha [email protected]


ReferencesAmerican Education al Research Association (AERA), American Psychological Association (APA), National Council on measurement in Education (NCME). (1999). Standards for educational and psychological testing. Washington, DC: Author.

Association of Test Publishers (ATP)/ Council of Chief State School Officers (CCSSO) (2013). Operational best practices for statewide large-scale assessment programs. Washington, DC: Author.

National Center for Education Statistics (NCES). (2013). Testing integrity symposium: Issues and recommendations for best practice. Washington DC: Institute of Education Sciences (IES). Washington DC: Institute of Education Sciences (IES). Retrieved from http://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2013454

National Council on Measurement in Education (NCME) (2012). Testing and data integrity in the administration of statewide student assessment programs. Madison WI: author.

Olson, J.F. & Fremer, J. (2013). TILSA test security guidebook. Washington, DC: Council of Chief State School Officers (CCSSO).