technology's roles in the nclb/ sol assessment dance david mott, reports online systems and...

Technology's Roles in the NCLB/SOL Assessment Dance

David Mott, Reports Online Systems and Tests for Higher Standards Presentation at the Virginia Society for Technology in Education (VSTE), Virginia Beach, February 27, 2007

Technology's Roles in the NCLB/SOL Assessment Dance

Technologists have vital roles to play in the current dance involving standards-based testing, the NCLB, AYP, school accreditation, and so on. Initially, technology was called upon to supply basic computer connectivity for internet-based, computer-delivered SOL tests. Having now largely accomplished this limited task, educators involved with technology are faced with a far more difficult task — supporting teachers and administrators in one of the highest profile sectors of the current era, assessment and testing. This support requires both IT knowledge and a broad understanding of the educational process itself. Automating parts of the assessment process can produce a very high return on investments in technological expertise and money. Properly deployed and implemented, IT-supported systems that gather and analyze test data, can give all levels of decision makers potent information. This session covers the specifics of how current technology can support data-informed instruction. We will talk about formative and summative assessment systems, the “right-sizing” of assessment technology, proper data collection and use, and the need for end-user training in all aspects of these new data based approaches. The context will be real-world, with specific examples using the Reports Online System with the Test for Higher Standards assessments. However, the general principals apply to any powerful automated assessment system.

About me My childern –

The animals –

Colin

JessicaDylan

Zack

Puck

Awk

About Me (so that you know where I’m coming from)

I’m an experimental psychologist. Was at the VA DOE for 17+ years, in what

is now called Assessment and Reporting. Supervisor of Test Development (and

psychometrics – test equating, etc.) Administrator of the VSAP (VA Student

Assessment Program, SAT-9) The division’s liason with the IT group In charge of procurements

How did I get into this business test publishing and scoring?

After I left the VA DOE, what I was going to do for the rest of my life?

Got together with Stuart Flanagan to do some SOL Mathematics tests.

Then came SOL English, Science, and History & Social Science tests.

Then we developed ROS so that teachers didn’t have to hand score tests.

Some Vital Roles for Technology and Technologists in the NCLB Era —

Supply connectivity to the LAN & Internet for: Administrative data bases Testing activities Student access Teachers’ uses

Training of: Administrators Teachers Students

Some Vital Roles for Technology and Technologists in the NCLB Era —

Recommend, buy, install, and support hardware

Recommend, buy, install, and support software

Plan for future directions for technology Participate in the production of technology

related products Help school personnel use the available

technology

Humor

Time flies like an arrow;

fruit flies like a banana.

Humor

Time flies like an arrow;

fruit flies like a banana.

Time flies like an arrow – but an arrow flies in only one direction! The point is that whatever happens to education in the present era, education will never go back to the way it was.

The point

A specific area: Data-Informed Decisionmaking

Why Data-Informed rather that Data-Driven?

Because data is not alive, it can’t be in the driver’s seat.

A person needs to be at the controls.

Data-Informed Decisionmaking — continued 1

Bus Schedules — Data % ontime lengths of route in miles # of students transported on bus

Bus Schedules — Questions with Answers Are the shorter routes ontime more often? What is the average student’s route distance? and so on.

Data-Informed Decisionmaking — continued 2

School Lunches — Data # of lunches served in a day Menu items SOL testing day or not a testing day

Bus Schedules — Questions with Answers Which menus are most popular? Are more lunches served on SOL testing days,

and is that dependent on the menu that day? and so on.

Data-Informed Instruction — Testing

Used as an example because it can be one of the heaviest and most mission-critical uses of technology – and because I know the most about it.

Also, there is much payback to be had from properly using testing data to inform instruction – which is, after all, the business we are in.

The ExaminationThe Examinationas shown by the Ghost of Testing Pastas shown by the Ghost of Testing Past

The ExaminationThe Examinationas shown by the Ghost of Testing Pastas shown by the Ghost of Testing Past

The Parts of an Assessment System

Student Information Database Test Generating System Test Presentation System Test Response Recording System Test Scoring System Test Reporting System Actionable Information

The Parts of an Assessment System Simple example, 7 parts – Teacher’s

Gradebook System


Choice Point

Assessment systems In general

ROS Online Testing

ROS specifically, with reports

?

The Parts of an Assessment System More complex example same 7 parts –

Reports Online System with content from Tests for Higher Standards


Assessment System – Reports Online System w/ Tests for Higher Standards

Student Information Database Extract from SASI or other student information system Name, ID, NCLB categories, Other info.

Test Generating System Test Presentation System Test Response Recording System Test Scoring System Test Reporting System Actionable Information

Assessment System – Reports Online System (ROS) w/ Tests for Higher Standards (TfHS)

Student Information Database Extract from SASI or other Name, ID, NCLB categories, Other info.

Test Generating System Paper test using MS Word & TfHS content Teacher-produced test District-produced benchmark test Textbook, coachbook, or other test


Paper test using MS Word & TfHS content Teacher produced test – printed, overhead,

oral, etc. District-produced benchmark test Textbook, coachbook, or other test

Test Presentation System Printed page, overhead projector,

whiteboard, oral, etc. Computer display


Test Presentation System Printed page, overhead projector, whiteboard, oral, etc. Computer display

Test Response Recording System Student written responses – notebook

paper, etc. Bubbled-in student answer sheets Responses on computer Responses on “clicker” (ROS doesn’t support this yet.)


Responses on computer Responses on “clicker” (ROS doesn’t support this yet.)

Test Scoring System ROS website servers w/ underlying database

One-time scoring vs. Active scoring (ROS is active.)

Cut points and variable item weights Active keys

Test Reporting System


Test Reporting System Reports on screen and on paper generated by

ROS website servers w/ underlying database

Item Analysis Report shows per-option response counts

Matrix Report compares performance on each item, standard, reporting category, and more

Progress Report outlines performance over time on multiple tests


Matrix Report compares performance on each item, standard, reporting category, and more

Progress Report outlines performance over time on multiple tests

Actionable Information A person (a teacher or administrator) needs to

combine the dry data in the reports with personal experience about students into real information.

Differentiated instruction needs to follow.

ROS Website

Transition to the ROS website

http://ros.tfhs.net/

http://rosworks.com/

For more information

ROS website – rosworks.com David Mott – [email protected]

866.724.7997

804.355.5489 TfHS website – www.tfhs.net

technology's roles in the nclb/ sol assessment dance david mott, reports online systems and...

Documents

test data

testing data

assessment technology

point slide

available technology

datainformed decisionmaking

new data

bus schedules data