26th annual management information systems [mis] conference february 14, 2013 washington, dc common...

26th Annual Management Information Systems [MIS]

ConferenceFebruary 14, 2013Washington, DC

Common Education Data Standards (CEDS) Supporting

Assessment Systems Development

Copyright © IMS Global and SIF Association

Overview• Need• What is AIF• Status• Demonstration Prototypes• Discussion and Questions


Created by Wordle.netCopyright © IMS Global and

SIF Association

Need• Standards– Confusing

• Assessment– RTTA and Consortia– Formative– Accountability

• Inform InstructionStock photo.

Assessment Lifecycle

ContentDevelopment

Pre-TestAdministration

TestAdministration

Scoring

Reporting

Post-TestAdministration

• Content and data reviews• Test form construction• Field testing• Item banking & statistics• Content exchange /

interoperability

•Planning & blueprinting•Item types• Content development &

universal design• Learning standard

alignment

• Administration planning & scheduling

• Registration, assignment,• Form sampling• Online infrastructure

readiness assessment• Pre-session planning

(paper / online) & setup• Alternate form

assignment

• Test form delivery• Platform (paper, online,

mobile) presentation• Item content & tools• Adaptive testing• Response collection• Proctoring controls• Form content security• Desktop security• Accessibility• Testing anomalies

• Computer scoring• Professional scoring• Algorithmic (AI) scoring• Portfolio scoring• Sub test / strand scoring

• Attemptedness• Performance levels• Scaling / norming• Growth scores• Range finding

AssessmentLife Cycle• Individual reporting

• Diagnostic reporting• Informing & personalizing

instruction• Performance on

standards• Dashboard / summary

reporting• Aggregation / disaggregation• Exchanging results / data

• Psychometric analysis• Equating• Score tables - scaling, norming• Performance levels / cut

scores• Field test analysis• Aligning results with

curriculum / instruction• Program and teacher

effectiveness

What is AIF?

• Systemic view of an assessment system and applications

• Demonstrates points of interoperability between applications within system

• Outlines the data model and transport


REGIONAL Systems

Assessment Platform Main Components

STATE Systems

Assessment Creation & Management System(ACMS)

LOCAL Systems

SIF

Assessment Delivery System(ADS)

Assessment Reporting System

(ARS)

APIPAPIP

SIF

APIP

SIF

APIP

Assessment Score Processing System

(ASPS)?

APIP

USDoE / EDFacts

Asse

ssm

ent P

latf

orm

Sub

-Com

pone

nts

REGION

STATE / ConsortiaSLDS, Accountability/AYP, Growth

Assessment Registration &

Administration System (ARAS)


LOCAL (LEA/School)SIS, LMS, Gradebook, Reporting System

AssessmentData

Warehouse (ADW)

3

2

6

9

1

AssessmentResults

OperationalReporting

(AROR)

10

11

4

Assessment Presentation &

Session Management System (APSMS)

Assessment Delivery System (ADS)

Assessment Reporting System (ARS)

16

Test Banking

Item Banking

PNP

PNP

Assessment Scoring Management

System (ASMS)

Assessment Analytics

System (AAS)

17 18 19

13

15 14

12

Assessment Scoring Analytics System

(ASAS)

20Assessment ScoreProcessing System (ASPS)

8

Adaptive Engine

21

Learning Standards 2222

Status

• Use cases outlined• Elements complete• Public review complete• Prototype demonstrations complete• Best practice document complete• Available on ceds.ed.gov/aif.aspx

Demonstration Pilots

• Logical and Physical Testing• Arrow 1• Arrow 10• Arrow 14

Arrow 1• Item Bank to Item Bank

USDoE / EDFacts

Ass

ess

me

nt P

latf

orm

Su

b-C

om

po

ne

nts

REGION






AssessmentData

Warehouse (ADW)

3

2

6

9

1

AssessmentResults


(AROR)

10

11

4





16

Test Banking

Item Banking

PNP

PNP


System (ASMS)


System (AAS)

17 18 19

13

15 14

12


(ASAS)


8

Adaptive Engine

21


Arrow 1 Diagram


Assessment Score Processing System (ASPS)


Assessment Registration & Administration System (ARAS)

Assessment Creation & Management System (ACMS)

Assessment Data Warehouse (ADW)

Assessment Results Operational Reporting (AROR)

Assessment Presentation & Session Management System (APSMS)

Test Banking

Item Banking

Assessment Scoring Management System (ASMS)

Assessment Analytics System (AAS)

Assessment Scoring Analytics System (ASAS)

10 – Item Bank Interoperability

Consortia Item Banks

State Item Banks

Local Item Banks

Item Bank Admin

Author orPsychometrician

Arrow 1 Test ScenarioScenario Description• The purpose of this scenario is to demonstrate assessment

items moving from an item authoring system to an item bank or from one item bank to another. This scenario covers only assessment items moving from an item authoring system to an item bank.

• A consortium has secured item authoring system A. This Item authoring system will not house the items; it is simply a means for item creation. The items will reside in item banking system B. Item authoring system A has created the assessment items and the editing and revision cycle has been complete. Item authoring system A is ready to send the items to item banking system B for storage and assessment instrument creation.


Test Scenario Components Required• Item authoring system• Item banking system

Pre-Condition• Assessment content (items, instruments, etc.) is ready to be

transferred in the sending item bank.

Post-Conditions• Content is now ready for use including reviews, edit,

extensions, assessment instrument creation, passing of instruments to delivery system, etc. Copyright © IMS Global and

SIF Association

Arrow 1 Test Scenario Cont’d


Arrow 1 Process Flow

Arrow 10• Assessment Registration

USDoE / EDFacts

Ass

ess

me

nt P

latf

orm

Su

b-C

om

po

ne

nts

REGION






AssessmentData

Warehouse (ADW)

3

2

6

9

1

AssessmentResults


(AROR)

10

11

4





16

Test Banking

Item Banking

PNP

PNP


System (ASMS)


System (AAS)

17 18 19

13

15 14

12


(ASAS)


8

Adaptive Engine

21



Arrow 10 Diagram









Test Banking

Item Banking




20 - Registration

Local SIS or LMS

State SLDS or Info System

Test Coordinator orTeacher

State Data Admin

Local Data Admin orTeacher

Consortia Info Systems

Consortia Data Admin


Arrow 10 Test ScenarioScenario Description• This scenario describes the necessary components for an assessment

registration. This scenario covers student demographic, teacher, hierarchy and PNP information.

• A consortium is ready to give an assessment. The student, school, LEA, PNP and teacher information have already been entered into the SIS or data warehouse. Assessment registration and administration system A pulls over all of the necessary information from the SIS or data warehouse B. The students are then assigned to a specific administration of an assessment. The registration and administration information is then passed from the assessment registration and administration system A to the assessment delivery system C.

• An alternate scenario would be that the PNP information is entered into the registration system after the information has been pulled over from the SIS or data warehouse.


Arrow 10 Test Scenario Cont’dTest Scenario Components Required• Student information system or data warehouse that houses all of

the necessary information for registration• Assessment registration and administration system• Assessment delivery system

Pre-Condition• All of the student, school, LEA and teacher information have been

entered into the SIS and/or Data Warehouse.• An administration of an assessment(s) has been identified.

Post-Conditions• The students are registered for a specific administration(s) of an

assessment.Copyright © IMS Global and

SIF Association

Arrow 14• Reporting System to Data Warehouse

USDoE / EDFacts

Ass

ess

me

nt P

latf

orm

Su

b-C

om

po

ne

nts

REGION






AssessmentData

Warehouse (ADW)

3

2

6

9

1

AssessmentResults


(AROR)

10

11

4





16

Test Banking

Item Banking

PNP

PNP


System (ASMS)


System (AAS)

17 18 19

13

15 14

12


(ASAS)


8

Adaptive Engine

21



Arrow 14 Diagram









Test Banking

Item Banking




40 – Distribute Results (local)

50 – Distribute Results (state)

Local SIS or LMS

State SLDS or Info System State Data Admin

Local Data Admin orTeacher

120 – Distribute Summary (local)

130 – Distribute Summary (State)

LEA Information or Reporting System

LEA Data Admin

Consortia Information or Reporting System

Consortia Data Admin


Arrow 14 Test Scenario

Scenario Description• The purpose of this scenario is to demonstrate sending

assessment results to the SEA from the Consortium administered assessment.

• The consortium has administered an assessment. The assessment results system A has received and compiled all information based upon the assessment. Assessment results system A compile, packages and disseminates the information to the SEA data warehouse B.


Arrow 14 Test Scenario Cont’dTest Scenario Components Required• Assessment results system• State data warehouse or reporting system

Pre-Condition• Assessment results are available and have been collected so

that summaries can be produced.

Post-Conditions• The state system has the results loaded.


Data Available by Entity• Assessment Family• Assessment• Assessment Form• Assessment Form Section• Assessment Form Subtest• Assessment Subtest Result• Assessment Performance Level• Assessment Item• Assessment Item Possible

Response• Assessment Item Response• Assessment Item Rubric• Assessment Participant Session• Learner Action• Assessment Administration• Assessment Registration• Assessment Session• Organization• Registration Accommodation• Person

• Achievement Evidence• Achievement• Learning Goal• Learning Assignment Learning

Resource• Learning Resource• Learner Activity• Learning Standard Document• Learning Standard Item• Competency Item_Competency Set• Competency Set• Learning Standard Item Grade Level• Assessment Levels for Which Designed• Assessment Form Subtest Levels for

Which Designed


Documentation

• Assessment Interoperability Framework Definitions and Requirements

• Use Cases for the Assessment Interoperability Framework

• Data Elements• Best Practices• Demonstration Prototype


Work Moving Forward

• Continue to evolve the technical standard within IMS and SIF respectively and reflect back to CEDS

• Implementation• Additional pilots• API and transport• Certification


ContactJill Abbott, CEOAbbott Advisor [email protected]

Rob Abel, CEOIMS Global [email protected]

Larry Fruth, CEOSIF [email protected]

All current documentation can be found at http://ceds.ed.gov/aif.aspx - Assessment Interoperability Framework (AIF)

mailto:[email protected]



http://ceds.ed.gov/aif.aspx

26th annual management information systems [mis] conference february 14, 2013 washington, dc common...

Documents

item authoring system

item banking system

item creation

assessment items

sif association

transport copyright

questionscopyright ims

data model