data loader user guide (2010-2011)

7/30/2019 Data Loader User Guide (2010-2011)

1/46

Data Loader

User Guide

2010-2011 School Year


2/46


3/46

iii | P a g e

Introduction

Data Driven Decisions for Academic Achievement

Data Driven Decisions for Academic Achievement (D3A2) is a long-term initiative focused on

developing the capacity of educators while improving instruction and student achievement.

D3A2 provides systemic access to timely data and educational resources aligned to Ohios

Academic Content Standards.

Data Warehouse & Data Tool

The D3A2 Data Warehouse and Data Tool provide the capacity for school districts in Ohio to

store, access, and analyze student course listings and performance data over multiple years.

District demographics data are supplied to the warehouse in the form of extracts fromthe eSIS and DASL student information systems or the EMIS reporting system. They in-

clude student demographic data, HQT teacher qualifications, and student-teacher-

course enrollment data.

Student performance data are provided in the form of results files distributed by Ohiostesting vendors following each administration of an Ohio Achievement Assessment

(OAA) or Ohio Graduation Test (OGT).

D3A2 Data Scrubber & Loader Utility

The D3A2 Data Scrubber and Loader is a web-based application that members of the State

Software Development Team (SSDT), Information Technology Centers (ITC), and school district

personnel use to prepare and load district demographics and student testing data into the D3A2

Data Warehouse. It combines features of the Data Scrubber and Loader applications, which used

to be stand-alone applications, under a single framework.

Implementation

The process of preparing and transferring data for upload to the D3A2 Data Warehouse is a

shared responsibility between ITCs and school districts.

ITCs are generally responsible for the following tasks associated with loading data into D3A2:

Extract and transfer demographics data from eSIS and DASL to their respective D3A2Drop Boxes in Oracle Collaboration Suite (OCS).

Monitor the extraction, transformation, and loading of data into the Data Warehouse. Work with the SSDT to identify and resolve issues which prevent files from loading into

D3A2.

Correct problems in raw demographics data to enable it to load successfully when appro-priate, or help district data entry staff correct errors in their SIS entries to correct faulty

demographics data.
http://www.d3a2.org/scrubber/http://www.d3a2.org/scrubber/http://www.d3a2.org/scrubber/


4/46


iv | P a g e

Create and manage ITC-administrator and district-user accounts in the D3A2 Scrubberand Loader.

Train district staff in the use of the Data Scrubber to prepare test results for loading. Validate scrubbing results and forward scrubbed testing data to the Loader. Report possible application bugs to the SSDT.

Districts are generally responsible for the following tasks associated with loading data into D3A2:

Maintain accurate data in eSIS and DASL. Correct data issues in eSIS or DASL that result in failed file loads. Upload and scrub raw vendor data in the D3A2 Data Scrubber (unless their ITC elects to

perform these activities on their behalf).

Verify the integrity of demographics and testing data in the D3A2 Data Tool applicationfollowing file uploads.

Report possible application bugs to their ITC.Loader ITC User Guide

The purpose of this guide is to document the use of the D3A2 Data Loader application and pro-vide details to support its use. It is made available to all Ohio school districts and Information

Technology Centers (ITC) by means of downloads from the project website.

Companion Documents

Additional documentation is available to support D3A2 data implementers.

D3A2 Data Scrubber User Guide D3A2 ITC Implementation Guide

These documents are both available for download from theSupport Materialspage on the

project website.

Project Website

More information about the D3A2 project, including the district implementation process, may be

discovered by visitingwww.d3a2.org.

Student Privacy

The names and course enrollments of students displayed in this guide have been fictionalized. No

actual students or district data is displayed in this document.
http://www.d3a2.org/uploads/docs/Support%20Materials/D3A2%20Data%20Scrubber%20User%20Guide%20(2010-2011).pdfhttp://www.d3a2.org/uploads/docs/Support%20Materials/D3A2%20Data%20Scrubber%20User%20Guide%20(2010-2011).pdfhttp://www.d3a2.org/uploads/docs/Support%20Materials/D3A2%20ITC%20Implementation%20Guide%202010-2011.pdfhttp://www.d3a2.org/uploads/docs/Support%20Materials/D3A2%20ITC%20Implementation%20Guide%202010-2011.pdfhttp://www.d3a2.org/support/http://www.d3a2.org/support/http://www.d3a2.org/support/http://c/Users/bell/Documents/D3A2/Documentation/Data%20Scrubber%20Manual/www.d3a2.orghttp://c/Users/bell/Documents/D3A2/Documentation/Data%20Scrubber%20Manual/www.d3a2.orghttp://c/Users/bell/Documents/D3A2/Documentation/Data%20Scrubber%20Manual/www.d3a2.orghttp://c/Users/bell/Documents/D3A2/Documentation/Data%20Scrubber%20Manual/www.d3a2.orghttp://www.d3a2.org/support/http://www.d3a2.org/uploads/docs/Support%20Materials/D3A2%20ITC%20Implementation%20Guide%202010-2011.pdfhttp://www.d3a2.org/uploads/docs/Support%20Materials/D3A2%20Data%20Scrubber%20User%20Guide%20(2010-2011).pdf


5/46


6/46


vi | P a g e

Appendix A: User Permissions

User Permissions .......................................................................................... 31

Appendix B: Notifications

Load Errors................................................................................................... 35

Prerequisite Errors ........................................................................................ 35

Purge Notification ........................................................................................ 35

Post Data Load ............................................................................................. 36

Threshold Exceeded ...................................................................................... 36


7/46

vii | P a g e

This page has intentionally been left blank.


8/46


9/46

1 | P a g e

Data Loader Basics

Features

The D3A2 Data Scrubber & Loader applications have been combined into a single framework to

provide Ohios Information Technology Centers (ITC) and school districts access to D3A2s com-

plete ETL process.

Testing File Scrubber Application

The D3A2 Data Scrubber provides a mechanism for Ohio School Districts to prepare their testing

results files for upload to the D3A2 Data Warehouse.

Refer to theD3A2 Data Scrubber User Guidefor complete instruction on how to access and usethe Data Scrubber application. It is available for download from theSupport Materials1 area of

the Project Website.

Loader Application

The D3A2 Data Loader provides an application to upload district demographics data and student

testing results into D3A2s longitudinal data system.

District Demographics Data

Student demographics, teacher HQT qualifications, enrollment data, and student-teacher-course

assignments are provided to the D3A2 system in the form of extract files created from eSIS,

DASL and Ohios EMIS reporting system.

These extracts consist of seven files for each school year: ID_SSID, DIST_INFO, BUILD_INFO,

STAFF_INFO, STU_INFO, STU_ENROLL, and STU_TEACH_COURSE. ITC personnel create

these extracts by running the correct procedure against the appropriate source.

1http://www.d3a2.org/support

Source of Demographics Data

1. 2010-2011 School Year: extracts from the eSIS and DASL student information systems.2. 2009-2010: School Year: extracts from the eSIS and DASL student information systems.

Note: eSIS extracts are not able to be created for historical reporting periods

3. 2008-2009 (and Prior) SchoolYear: extracts created from the EMIS database.
http://www.d3a2.org/uploads/docs/Support%20Materials/D3A2%20Data%20Scrubber%20User%20Guide%20(2010-2011).pdfhttp://www.d3a2.org/uploads/docs/Support%20Materials/D3A2%20Data%20Scrubber%20User%20Guide%20(2010-2011).pdfhttp://www.d3a2.org/uploads/docs/Support%20Materials/D3A2%20Data%20Scrubber%20User%20Guide%20(2010-2011).pdfhttp://www.d3a2.org/supporthttp://www.d3a2.org/supporthttp://www.d3a2.org/supporthttp://www.d3a2.org/supporthttp://www.d3a2.org/uploads/docs/Support%20Materials/D3A2%20Data%20Scrubber%20User%20Guide%20(2010-2011).pdf


10/46


2 | P a g e

Note: district demographics files must be loaded into the D3A2 Data Warehouse in chronologi-

cal order according to school year, sometimes referred to as reporting period.

More Info: districts should decide how far back they intend to scrub testing results and request

their ITC to create and load district demographics for all school years for each of these years.

District demographics data should be sent to the D3A2 Loader for processing once for each prior

school year and at regular periods throughout the current school year. Providing regular uploads

of current year demographics data ensures that D3A2s re-rostering features are kept up-to-date.

More Info: ITCs are advised to submit current year district demographics to the Loader as soon

as participating districts initial class lists are available in theirstudent information systems and

then at least once per quarter thereafter.

Note: the D3A2 extract procedure for eSIS and DASL may be set up to run at scheduled inter-

vals. ITCs are asked not to send data to the D3A2 Loader more frequently than once per monthunless there has been significant student movement or data entry corrections within a districts SIS.

General instruction on how to create district demographic file extracts from the EMIS database

are provided in theD3A2ITC Implementation Guide. eSIS and DASL users should refer to their

SIS documentation for additional information.

Oracle Collaboration Suite

Oracle Collaboration Suite (OCS) is used by ITCs to securely transfer district demographics files

to the Loader staging area. Each ITC has its own username and D3A2 Drop Box in OCS into

which district demographics files are deposited for automatic upload and processing.

More information about OCS is provided in later sections of this Guide, as well as in the D3A2 ITC

Implementation Guide.

Note: the automatic loading of files deposited into the D3A2 Drop Boxes is a new feature of the

D3A2 implementation process beginning with the 2010-2011 school year.
http://www.d3a2.org/support/http:/www.d3a2.org/uploads/docs/Support%20Materials/D3A2%20ITC%20Implementation%20Guide%202010-2011.pdfhttp://www.d3a2.org/support/http:/www.d3a2.org/uploads/docs/Support%20Materials/D3A2%20ITC%20Implementation%20Guide%202010-2011.pdfhttp://www.d3a2.org/support/http:/www.d3a2.org/uploads/docs/Support%20Materials/D3A2%20ITC%20Implementation%20Guide%202010-2011.pdfhttp://www.d3a2.org/support/http:/www.d3a2.org/uploads/docs/Support%20Materials/D3A2%20ITC%20Implementation%20Guide%202010-2011.pdfhttp://www.d3a2.org/support/http:/www.d3a2.org/uploads/docs/Support%20Materials/D3A2%20ITC%20Implementation%20Guide%202010-2011.pdfhttp://www.d3a2.org/support/http:/www.d3a2.org/uploads/docs/Support%20Materials/D3A2%20ITC%20Implementation%20Guide%202010-2011.pdfhttp://www.d3a2.org/support/http:/www.d3a2.org/uploads/docs/Support%20Materials/D3A2%20ITC%20Implementation%20Guide%202010-2011.pdfhttp://www.d3a2.org/support/http:/www.d3a2.org/uploads/docs/Support%20Materials/D3A2%20ITC%20Implementation%20Guide%202010-2011.pdfhttp://www.d3a2.org/support/http:/www.d3a2.org/uploads/docs/Support%20Materials/D3A2%20ITC%20Implementation%20Guide%202010-2011.pdfhttp://www.d3a2.org/support/http:/www.d3a2.org/uploads/docs/Support%20Materials/D3A2%20ITC%20Implementation%20Guide%202010-2011.pdfhttp://www.d3a2.org/support/http:/www.d3a2.org/uploads/docs/Support%20Materials/D3A2%20ITC%20Implementation%20Guide%202010-2011.pdfhttp://www.d3a2.org/support/http:/www.d3a2.org/uploads/docs/Support%20Materials/D3A2%20ITC%20Implementation%20Guide%202010-2011.pdf


11/46

Loader Basics

3 | P a g e

Testing Vendor Files

Vendors of the Ohio Achievement and Graduation tests provide districts with electronic copies of

results files following each test administration. These files may be prepared for upload to the

D3A2 Data Warehouse and consist of thegeneral (student scores) and item analysis files.

These files must be processed using the D3A2 Data Scrubber prior to upload to the Data Ware-house. Detailed instruction in how to access and process these files is provided in theD3A2 Data

Scrubber User Guide.

More Info: unlike district demographics files which are automatically picked up and processed

from the ITCs OCS Drop Boxes, testing files must be uploaded to the D3A2 Data Scrubber & Loader

application manually. Information about this process is provided in detail in the D3A2 Data Scrub-

ber User Guide.
http://www.d3a2.org/uploads/docs/Support%20Materials/D3A2%20Data%20Scrubber%20User%20Guide%20(2010-2011).pdfhttp://www.d3a2.org/uploads/docs/Support%20Materials/D3A2%20Data%20Scrubber%20User%20Guide%20(2010-2011).pdfhttp://www.d3a2.org/uploads/docs/Support%20Materials/D3A2%20Data%20Scrubber%20User%20Guide%20(2010-2011).pdfhttp://www.d3a2.org/uploads/docs/Support%20Materials/D3A2%20Data%20Scrubber%20User%20Guide%20(2010-2011).pdfhttp://www.d3a2.org/uploads/docs/Support%20Materials/D3A2%20Data%20Scrubber%20User%20Guide%20(2010-2011).pdfhttp://www.d3a2.org/uploads/docs/Support%20Materials/D3A2%20Data%20Scrubber%20User%20Guide%20(2010-2011).pdf


12/46


13/46

1 | P a g e

Accessing the Data Scrubber & Loader

Login Accounts

The Loader login screen is located at

http://Loader.d3a2.org.

Login accounts are required to use the D3A2

Scrubber & Loader Application. Each ITC is

assigned an initial userID and password and then

begins creating and managing accounts for their

own organization and school districts.

Who Should Use the Scrubber & Loader?Depending upon policies established by each ITC, districts are required to participate to a greater

or lesser extent in the scrubbing process. The policy of most ITCs is that, once trained, districts

are required to log in to the Data Scrubber to scrub their test return files independently. The ITC

provides support as needed to help districts through any difficulties they experience with the

scrubbing process.

The district staff member(s) assigned to the task of scrubbing testing vendor data need to have

knowledge of and administrative access to the districts student information system and EMIS

data. They must be comfortable using data systems. These staff often serve in the role of EMIScoordinators, technology coordinators, or curriculum directors within their districts.

User Types

The Scrubber supports three types of user accounts: District, ITC, and Administrator. Each type

of account provides differing levels of access to features within the Loader as described in Ap-

pendix A of this Guide.

Figure 1: Scrubber & Loader Login Screen
http://www.d3a2.og/scrubberhttp://www.d3a2.og/scrubberhttp://www.d3a2.og/scrubber


14/46


2 | P a g e

Creating User Accounts (Step by Step)

The D3 A2 Operations Group assigns each ITC an initial username and password for the Loader

application. ITCs are able to use this account to create additional ITC and district-level accounts.

1.) Log in to the application using an ITC-levelaccount

2.) Select User Maintenance from the main

menu.

3.) Click .

4.) Enter the users information.

Note: ITC staffmay only select District

or ITCin the User Role selection field.

Note: when creating ITC-level accounts,

be sure to select a school districtin the Dis-

trict field, rather than the ITC.

More Info: the District selection field only

dislays districts that are registered to the ITC

users organization. If a district is not showing

as it should, the ITC should issue a heldesk

ticket to the SSDT.D3A2.SUPPORT request

area asking that the district be reassigned.

Caution:assigning an ITC-level account to an ITC rather than a school district in the District

selection field will result in the user not being able to view district-user accounts or data as they

should. This can only be corrected by submitting a help desk ticket to the SSDT.D3A2.SUPPORT

request area in CA-Unicenter.

5.) Click .

Figure 2: Create New User

Figure 3: User Details Screen
http://upload.wikimedia.org/wikipedia/commons/thumb/2/24/Warning_icon.svg/420px-Warning_icon.svg.png


15/46

Loader Basics

3 | P a g e

Managing User Accounts (Step-By-Step)

Recover Forgotten Passwords

ITC and district-level account holders are able to

reset forgotten passwords from the Loader login

page.

1.) Click on Forgot Your Password? Click here to

resetfrom the Login screen.

2.) Enter valid email address to which the account

is registered. After you click Reset Password the

Scrubber will send a temporary password to your

email account.

Figure 4: Forgot Password

Figure 5: Reset Password


16/46


4 | P a g e

Changing Passwords

All users of the Loader should personalize their password the first time they log into the Scrubber

& Loader application.

Caution: the Scrubber exposes sensitive student information. Always select a complex pass-

word and do not share your password with anyone else.

3.) Click on the Password maintenance link shown in the page header.

4.) Enter and confirm a new password,

then click .

5.) Verify that the Password changedmessage is displayed.

Figure 8: Password Change Confirmation

More Info: Scrubber passwords should be at least 6 characters in length and include at least one

number.

Figure 6: Change Password Link

Figure 7: Password Page


17/46

Loader Basics

5 | P a g e

Edit User Accounts

ITC-level Loader accounts are able to view and manage accounts from their local user base.

6.) Log in to the application using an ITC-levelaccount

7.) Select User Maintenance from the menu.

8.) Browse for an account to edit, or search, and

click the edit ( ) icon to access the User

Details page.

More Info: ITC-level account holders only

see users belonging to their organizations dis-

tricts on the User Maintenance page.

Note: if a user who is known to have an

existing account is not appearing on the User

Maintenance page it is likely that they have been

assigned to another ITCs district. Submit a

helpdesk ticket to the SSDT.D3A2.SUPPORTrequest area in Unicenter asking to have them

reassigned to a different district and/or have their

district assigned to a different ITC.

9.) Edit the users details as necessary.

10.) Click .

Figure 9: Edit User Details Link

Figure 10: Edit User Details Page


18/46


6 | P a g e

Main Application Pages

The various tasks associated with loading data into the D3A2 Data Warehouse are accomplished

using the Loaders three main pages: Loader Activity, Status Details, and Unmet Data Prerequi-

sites pages.

Loader Activity Page

The Loader Activity page provides a snapshot

of recent activity in the D3A2 Loader and

provides access to score cards for demograph-

ic and testing file loads.

Status Details PageThe Status Details page provides detailed

information about what has been loaded for a

given school district and/or what is currently

being processed by the Loader. This page

allows users to:

research which files have been loaded andany errors associated with them;

identify files that can not be processed dueto critical errors;

release testing files that have been for-warded to them for review by district

scrubber personnel; and

observe loading progress for files that havebeen enqueued in the Loader.

Unmet Load Prerequisites Page

The Unmet Load Prerequisites page provides

an area to view unmet prerequisites which are

preventing files to be acted upon by the Loader

after they have made their way into the

processing area.

Fi ure 11: Loader Activit Pa e

Figure 12: Status Details Page

Figure 13: Unmet Load Prerequisites Page


19/46

Loader Basics

7 | P a g e

Navigating the Loader Pages

Navigating between the Loader pages is accom-

plished by hovering over the Navigation Menu

and making a selection from the drop down.

Using the Scoping Tool (Step-By-Step)

ITCs are initially presented with loading activities and status for all of their member districts when

they log into the Loader.

The Scoping Tool allows ITC staff to filter the display to show data for just one district.

1.) Click on or

to open the Scoping Tool.

2.) Select a School District , Year , or both from

the prompts.

Note: ITC staff are only presented districts

that are assigned to their own organizations when

they use the Scoping Tool.

3.) Click to register the selection.

Figure 14: Data Loader Navigation Menu

Figure 15: Access the Scoping Tool

Figure 17: Verify Scoping Selection

Figure 16: District & Year Selection


20/46


21/46

9 | P a g e

Loading District Demographics Files

ITC support staff participate in the loading of test results files by performing the following tasks:

Extracting demographic files from an EMIS database or the student information system; Uploading demographics files to their organizations D3A2 Drop Box in Oracle Collabora-

tion Suite (OCS);

Monitoring the progress of demographic file loads using the Status Details page; Evaluating the results of successful loads using the Status Details page; and Responding when errors in the raw extracts data prevent the normal processing of data.

Uploading District Demographics Files to

OCS (Step-By-Step)1.) Extract district demographics data from an

appropriate source.

2.) Use the ITCs username and password to

access OCS:.

a.) use Internet Explorer 7 to log in to

https://wildfire.nwoca.org/content/dav; or

b.) map a network drive in MS Windows My

Computer tohttps://wildfire.nwoca.org/content/.

More Info: operational support for OCS

may be requested by issuing a helpdesk to the

SSDT.D3A2.SUPPORT request area in CA-

Unicenter.

2.) Upload district demographics data to the

ITCs D3A2 Drop Box in OCS to initiate the

loading.

More Info: do not copy the ID_SSID infor-

mation file to the Drop Box in OCS.

Figure 18: OCS D3A2 Drop Boxes

Figure 19: Uploaded Files
https://wildfire.nwoca.org/content/davhttps://wildfire.nwoca.org/content/davhttps://wildfire.nwoca.org/content/dav


22/46


10 | P a g e

3.) The Loader automatically picks up and processes the new district demographics files into its

Processing Files list on the Status Details page.

Note: it is possible to send individual files for load in order to recover from a prior critical error.

For example, a Student Enrollment File may be dropped into the Drop Box for processing. However,

this can result in unexpected results. ITC staff should discuss specific situations with the D3A2

Operations Group that may require processing files individually before attempting this procedure.

More Info: file extracts from prior years should be loaded once into the Data Warehouse, whe-

reas extracts from the current school year should be loaded at least quarterly and/or whenever

significant changes have been made in the student information system.

Caution:district demographics loads must be loaded into the Data Warehouse in chronological

order according to school year. ITC staff must first determine how far back their districts intend to

scrub testing data before sending demographics files to the Loader. This will help ensure that ex-

tracts from all necessary school years are processed through the Loader in the proper order.

District Demographics Files

D3A2 demographics loads consist of six files:

Load Type Filename File Type

1 DIST_INFO_XXXXXX_xxL.csv District Information

2 BUILD_INFO_XXXXXX_XXL.csv Building Information

3 STU_INFO_XXXXXX_XXL.csv Student Information

4 STAFF_INFO_XXXXXX_XXL.csv Staff Information

5 STU_ENROLL_XXXXXX_XXL.csv Student Enrollment

6 STU_TEACH_COURSE_INFO Student Teacher Course Information

The Loader processes each of these files sequentially in order of type.


23/46


11 | P a g e

Monitoring Demographic File Loads (Step- By-Step)

The Status Details page is used to monitor the progress of district demographic file loads.

1.) Log into the Data Loader application using an ITC-level account.

2.) Navigate to the Status Details page.

3.) Use the Scoping Tool to narrow the view

to a single district.

4.) Check to make sure that there are no critical errors reported by clicking on the Status Details

page Files in Error list.

5.) Select the Released File list by clicking on and observe that all six district demo-

graphic files have been transferred from the ITCs Drop Box to the Loader.

Figure 20: Navigate to Status Details



24/46


12 | P a g e

6.) The Loader will process all files sequentially in order of load type as indicated by the

Processing or Waiting to be Processed progress indicators .

7.) Observe that the testing files are eventually processed out of the list into

.

More Info: ITC and district users both receive notification by email when files have successfully

been loaded into the data warehouse. Notification occurs by email following the next weekend run

of the D3A2 post data load procedure. The email message includes a directly link to a scorecard for

the load.

More Info: both the ITC and district users receive notifications whenever (1) the minimum

threshold of rows in error is exceeding during loading of a file (typically this 25% or more), or (b) a

critical error is encountered with a file that stalls processing.

Note: Appendix B of this Guide provides a summary of Loader email notifications.

Load Prerequisites

Rule Sequence Rule Description

10 District must be authorized to be included in D3A2 processing. 20 District must have no unresolved load errors.

30 District must have no other loads in progress.

40 Sequential load types for district must be completed for demographics

files.

50 District must have no prior load types pending.

60 Multiple instantiations for a load type for a given district must be

processed in the order in which they were received.

70 District must have no loads from prior school years pending.

80 Processing for prior school year demographics must be completed for

the given district.


25/46


13 | P a g e

Evaluating Results of Successful Demographics Loads (Step-By-Step)

The Status DetailsCompleted Files list is used to evaluate the relative completeness of successful

file loads.

More Info: in this Guide, loads are said to be successfulwhen they have been processed

through to the Data Warehouse. This terminology does not consider how may rows were excluded

from loading due to failed validations.

1.) Open the Status DetailsCompleted Files list by clicking .

Note: each of the demographic file loads will show multiple times since they are typically

processed through the Loader more than once per school year. Scroll to the Start Time and End

Time columns to find the most recently loaded version of the file you are interested in researching.

2.) Check to see that an appropriately high percentage of rows were either inserted or updatedversus being in error.

Figure 22: Completed Loads List

Caution:if a very large percentage of rows are in error, then the data that is stored in the Data

Warehouse is not accurate

Note: create a CA-Unicenter ticket assigned to the SSDT.D3A2.SUPPORT request area asking

that the D3A2 Operations Group evaluate and purge a file from the Data Warehouse which has not

loaded properly or displays a large proportion of failed to successful rows.


26/46


14 | P a g e

3.) If necessary, click on the links under the Rows in Errorto display a row-by-row account of

records that were not uploaded to the Data Warehouse and the reasons they failed.

Figure 23: Failed Validation (Rows in Error) Report

Note: the Failed Validation Report is valuable in diagnosing the cause for failed records, as well

as researching the cause of missing data in the D3A2 Data Tool.

More Info: clicking on the links under the Rows Inserted column displays a Load Rule

Processing Report which the D3A2 Operations Team uses to evaluate failed loads.

Correcting Errors in the Extract Files

Errors found in extract files typically originate from inaccurate entries in the districts student

information systems. Severe problems in extract files will trigger fatal errors in the Loader which

prevents them from being processed. Such errors are typically resolved by correcting data in the

SIS, re-extracting the D3A2 demographics files, and re-processing through the Loader.

The D3A2 Operations Staff can assist in diagnosing the cause of load errors. Occasionally errors

can be repaired directly in an extract file to allow it to pass the Loaders validation rules.


27/46


28/46


16 | P a g e

5.) Use the Scoping Tool to narrow the view to

the correct district.

6.) Navigate to the file list and ob-

serve the high-level scrubbing results for the

test file(s).

Note: hovering over the testing file status

indicator instantly provides a display of the

numbers of unmatched teacher crosswalk

records, unmatched student crosswalk records,

and null teacher IDs in the testing data.

7.) If the number of unresolved IDs warrants

closer inspection, navigate to the Student

Scores Data page and thoroughly review the

testing data in the Scrubber.

8.) If additional scrubbing is necessary, click

from the Student Scores Data or

Item Analysis Data page and then Contin-

ue to back the file out of the Loaders Files

on Holdlist or .

Figure 27: Testing Files on "Hold" Status

Figure 28: Reviewing Scrubbing Results


Figure 29: Confirm Testing File Recall


29/46

Loading Testing Files

17 | P a g e

9.) Release testing file(s) for processing by

check marking them and clicking

while viewing Status

DetailsFiles on Holdlist.

More Info: two files are forwarded to the Files on Hold list when districts finalize and forward

student scores filesTest Scores andTest Part Scores.

10.) The Loader will move the files from the list to the list.

11.) Check to make sure that there are no critical errors reported on the Status Details pages

Files in Errorlist .

Caution:the Loader will not process any additional files for a district when a previously loaded

file has been held up in the Files in Error list. ITC staff must clear any critical errors before loading of

additional files can proceed.

Issues Affecting how Unresolved Testing Data is Processed

Unresolved testing records are handled differently during loading depending upon the type

of problem:

Records with null teacher IDs are resolved to the unknown teacher. Records with null student IDs are excluded from loading into the warehouse. Records with unmatched student or teacher crosswalk entries are excluded from loading

into the warehouse

Note: this could represent a significant amount of data if the unmatched teacher taught a

large number of students.

Records with unknown teacher IDs, i.e. IDs that are not represented in the districts EMISor SIS data, are resolved to the unknown teacher.

Figure 30: Release Testing Files for Processing


30/46


18 | P a g e

Monitoring Testing File Loads (Step- By-Step)

The Status Details page is used to monitor the progress of testing file loads.



3.) Use the Scoping Tool to narrow the view

down to a single district.

4.) Check to make sure that there are no critical errors reported on the Status Details pages Files

in Errorlist .

5.) Click on and observe that

the testing files have transferred from the

Student Scores Dataor Item Analysis

Datapage to the Loader.

6.) The Loader automatically begins processing all files sequentially by order of load type as

indicated by the Processing or Waiting to be Processed progress indicators. Processing will

occur automatically unless unmet prerequisites or critical errors prevent files from being acted

upon.

Figure 31: Navigate to Status Details


Figure 33: Monitor Progress Using Status Details


31/46


19 | P a g e

7.) Observe that the testing files are eventually processed out of the list into

.

More Info: ITC and district users both receive notification by email when files have successfully

been loaded into the data warehouse. Notification occurs following the next weekly run of the D3A2

post data load procedure and provides a link to a scorecard for the load.

More Info: both the ITC and district users receive notifications whenever (1) the minimum

threshold of rows in error is exceeding during loading of a file (typically 25% or more), or (b) a critical

error is encountered with the file that stalls processing.

Load Prerequisites

Rule Sequence Rule Description

10 District must be authorized to be included in D3A2 processing.

20 District must have no unresolved load errors.

30 District must have no other loads in progress.

40 Sequential load types for district must be completed for demographics

files.

50 District must have no prior load types pending.

60 Multiple instantiations for a load type for a given district must be

processed in the order in which they were received.

70 District must have no loads from prior school years pending.

80 Processing for prior school year demographics must be completed for

the given district.


32/46


20 | P a g e

Evaluating Results of Successful Testing File Loads (Step-By-Step)

The Status DetailsCompleted Files list is used to evaluate the relative completeness of successful

file loads.

1.) Open the Status DetailsCompleted Files list by clicking .

2.) Check to see that an appropriately high proportion of rows were either inserted or updated

versus being in error.

Figure 34: Completed Loads List

Caution:if a very large percentage of rows are in error, then the data contained in the Data

Warehouse is not accurate

Note: create a CA-Unicenter ticket assigned to the SSDT.D3A2.SUPPORT request area asking

that the D3A2 Operations Group evaluate and purge a file from the Data Warehouse which has not

loaded properly or displays a large proportion of failed to successful rows.


33/46


21 | P a g e

3.) If necessary, click on the links under the Rows in Errorto display a row-by-row account of

records that were not uploaded to the Data Warehouse and the reasons they failed.

Figure 35: Failed Validation (Rows in Error) Report

Note: the Failed Validation Report is valuable in diagnosing the cause for failed records, as well

as for determining exactly which records were excluded from the load.

More Info: clicking on the links under the Rows Inserted column displays a Load Rule

Processing Report which the D3A2 Operations Team uses to evaluate failed loads.

Correcting Errors in Testing File Loads

Errors found in testing files typically originate from the Scrubber. Such errors must be reported

to and resolved by the D3A2 Operations Staff.

More Info: submit a CA-Unicenter ticket to the SSDT.D3A2.SUPPORT request area to report

critical errors preventing the loading of testing files.


34/46


35/46

23 | P a g e

Using the Loader Activity and

Unmet Load Prerequisites Pages

The Loader Activity and Unmet Load Prerequisitespages provide at a glance information to

ITCs about Loader activity and status.

The Loader Activity page shows a listing of districts most recent loading activity. It also

provides access to a Score Cardreport for each load. The Score Carddetails the number of rows

in the raw data, the number not loaded, and the number loaded in each file.

The Unmet Prerequisites report provides information about files that have queued for loading

but cannot be acted upon due to unmet prerequisites.

Use the Loader Activity Page to Review Load Score Sheets (Step-By-Step)

The Status Details page is used to monitor the progress of district demographic file loads and

view load score cards.



3.) If necessary, use the Scoping Tool to

narrow the view down to a specific district.

Figure 36: Navigate to Loader Activity



36/46


24 | P a g e

4.) Review all recent Loading activity for your

districts.

5.) Click on a file to open its Score Sheet.

6.) Review the score sheet from the latest demo-

graphics or testing file load or ...

Figure 39: Click on Row to Access Score Sheet

Figure 40: Review Latest Score Card Information

Figure 38: Review Recent Loading Results


37/46


25 | P a g e

7. Click on a blue demographic file link

(when available) to expose complete

loading history for the selected file

during any academic reporting period.

Note: score card results for multiple demographics loads performed during a given academicperiod are able to be listed in chronological order on this report.

Note: links are not provided if a demographic file has only been loaded once for the academic

term.

More Info: the Added for Groups column only applies to general (student scores) testing files.

It indicates how many scores were duplicated to create standards reporting groups in the subjects of

Science and Social Studies.

8.) Click on a number link under the Not

Loadedcolumn to display the Failed Valida-

tion report for the load.

Figure 41: Analyze All Score Card Information

Figure 42: Failed Validation Report


38/46


39/46

Appendix A: User Permissions


40/46


41/46

User Permissions

The following table shows all permissions based upon user role.

Menu Page Subpage Function User Role

District ITC Administrator

Password Change Password Yes Yes Yes

Administration Access Data User Maintenance Create New User Yes YesAdministration Access Data User Details Delete User Yes Yes

Administration Access Data User Details Edit User Yes Yes

Administration Access Data District Maintenance Create New District Yes

Administration Access Data District Details Edit District Yes

Administration Access Data District Details Delete District Yes

Administration Loader Control Load Rule Maint Create Load Rule Yes

Administration Loader Control Load Rule Maint Edit Load Rule Yes

Administration Loader Control Load Rule Maint Disable Load Rule Yes

Administration Loader Control Prerequ Rule Maint Create Prerequisite Yes

Administration Loader Control Prerequ Rule Maint Edit Prerequisite Yes

Administration Loader Control Prerequ Rule Maint Delete Prerequisite Yes

Administration Loader Control Prerequ Rule Maint Disable Prerequisite Yes

Administration Loader Control Load Type Maint Edit Load Type Yes

Administration System Control Disable Sched Job Yes

Administration System Control Update Base URL Yes

Administration User Statistics View Activity Stats Yes

Administration Data Mapping Test Part Mapping Create New Mappings Yes

Administration Data Mapping Test Part Mapping Edit Mappings Yes

Administration Data Mapping Test Part Mapping Delete Mappings Yes

Administration Data Mapping Vendor Layout Map View Layout Maps Yes

Data Scrubber Vendor File Upload Upload Test File Yes Yes Yes

Data Scrubber Vendor File Upload Start Scrubbing Yes Yes Yes

Data Scrubber Stud Score Data Finalize & Forward Yes Yes Yes

Data Scrubber Stud Score Data Recall Yes Yes Yes

Data Scrubber Stud Score Data View Verification Yes Yes Yes

Data Scrubber Stud Score Data Teach X-Walk Match Edit Crosswalk Record Yes Yes Yes

Data Scrubber Stud Score Data Teach X-Walk Match Regenerate X-Walk Yes Yes Yes

Data Scrubber Stud Score Data Teacher X-Walk Mat Run BMW Yes Yes Yes

Data Scrubber Stud Score Data Edit Stud Score Dtail Edit Score Record Yes Yes Yes

Data Scrubber Stud Score Data Edit Stud Score Dtail Delete Score Record Yes Yes Yes

Data Scrubber Stud Score Data Advance Rpt Filter Mass Update Records Yes Yes Yes

Data Scrubber Stud Score Data Stud X-Walk Match Edit Crosswalk Record Yes Yes Yes

Data Scrubber Stud Score Data Stud X-Walk Match Regenerate X-Walk Yes Yes Yes

Data Scrubber Stud Score Data Stud X-Walk Match Upload ID_SSID File Yes Yes Yes

Data Scrubber Item Analysis Data Teach X-Walk Match Edit Crosswalk Record Yes Yes Yes

Data Scrubber Item Analysis Data Teach X-Walk Match Regenerate X-Walk Yes Yes Yes

Data Scrubber Item Analysis Data Teacher X-Walk Mat Run BMW Yes Yes Yes

Data Scrubber Item Analysis Data Edit Stud Score Dtail Edit Score Record Yes Yes Yes

Data Scrubber Item Analysis Data Edit Stud Score Dtail Delete Score Record Yes Yes Yes

Data Scrubber Item Analysis Data Edit Item Detail Edit IA Record Yes Yes Yes

Data Scrubber Item Analysis Data Edit Item Detail Delete IA Record Yes Yes Yes

Data Loader Loader Activity Score Sheet View Score Sheet Yes Yes Yes

Data Loader Loader Activity Score Sheet View Fail Valid Rpt Yes Yes Yes

Data Loader Status Details Completed Purge Files Yes

Data Loader Status Details Completed View Fail Valid Rpt Yes Yes Yes

Data Loader Status Details Completed View Fail Process Rpt Yes Yes Yes

Data Loader Status Details Errors Purge Files Yes

Data Loader Status Details Errors Reprocess Files Yes Yes

Data Loader Status Details Hold Release Files Yes Yes

Data Loader Status Details Released View Unmet Prreqs Yes Yes Yes


42/46


43/46

Appendix B: Notifications


44/46


45/46

Load Errors

Recipients:[email protected],[email protected], ITC Users

Subject: D3A2 Loader Error IRN

An error was encountered while loading for IRN , school year .

The error reported was :

The file ID is :

Prerequisite Errors

Recipients:[email protected],[email protected],

ITC Users, District Users

Subject: D3A2 Prerequisite Notice IRN

could not be loaded for IRN , school

year

A critical prerequisite failed. The reported prerequisite was :' ;

-

A hold has been placed on the file.

The file ID is :

Purge Notification



Subject: D3A2 Purge Notice IRN

for IRN , school year

has been purged from the Loader and, if applicable, the D3A2

warehouse.
mailto:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]


46/46


Post Data Load



Subject: Score Sheet Update

You have new data in D3A2. Please review the activity report and therelated scoresheets.

http://

Threshold Exceeded



Subject: D3A2 Loader Error IRN

An error was encountered while loading Student Information for IRN 044875, schoolyear 10L.

The error reported was :

Oracle Exception encountered

Error code

Error text ORA-12899:

".""."" (, )

The file ID is :
mailto:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]