assessment and surveys (nasop)...national assessment and surveys online program (nasop) development...

National Assessment and Surveys Online Program (NASOP) Development Study 2014 Appendix 1 Business Requirements – Testlet and Test Creation, Review and Delivery System

Version 1.0

Dated 5‐12‐2013

ii National Assessment and Surveys Online Program (NASOP) Development Study 2014 Appendix 1 Business Requirements – Testlet and Test Creation, Review and Delivery System,

Contents

1. OVERVIEW ................................................................................................................... 3

1.1. BACKGROUND ........................................................................................................................... 3 1.2 THIS DOCUMENT ....................................................................................................................... 3

2. BUSINESS PROCESSES .................................................................................................. 4

2.1 LISTING .................................................................................................................................... 4

3. BUSINESS REQUIREMENTS ........................................................................................... 5

3.1 NASOP 2014 DEVELOPMENT STUDY ‐ STAGES....................................................................... 5 3.1.1 Identity .................................................................................................................................. 5 3.1.2 Security .................................................................................................................................. 5 3.1.3 Standards .............................................................................................................................. 5 3.1.4 Auditing and Editing .............................................................................................................. 5

3.2 TD1 – INGEST TEST ITEM ....................................................................................................... 6 3.2.1 Business Requirements ......................................................................................................... 6

3.3 TD3 – CONSTRUCT TEST ......................................................................................................... 9 3.3.1 Business Requirements ......................................................................................................... 9

3.4 PTA2 – READINESS FOR TEST AT THE LOCAL LEVEL .............................................................. 10 3.4.1 Business Requirements ....................................................................................................... 10

3.5 PTA3 – REGISTER FOR TEST .................................................................................................. 11 3.5.1 Business Requirements ....................................................................................................... 11

3.6 TA1 – DELIVER TEST ............................................................................................................. 14 3.6.1 Business Requirements ....................................................................................................... 15

3.7 MS1 – MARK AND SCORE TESTS ............................................................................................ 19 3.7.1 Business Requirements ....................................................................................................... 20

3.8 NON‐FUNCTIONAL REQUIREMENTS ............................................................................................. 21

4. SUPPORTING DOCUMENTS ........................................................................................ 22

4.1 METADATA LISTING ............................................................................................................. 22

National Assessment and Surveys Online Program (NASOP) Development Study 2014 Appendix 1 Business Requirements – Testlet and Test Creation, Review and Delivery System, 3

1. Overview 1.1. Background

The National Assessment and Surveys Online Program (NASOP) is designed to deliver national assessments and surveys online. The program has three elements:

a) development, testing and delivery of a national online School Opinion Survey for deployment in 2013 at jurisdictional discretion (Element 1),

b) development, testing and delivery online of the 2013 National Assessment Program Civics and Citizenship sample assessment (NAP‐CC) (Element 2), and

c) development and testing of a National Online Assessment Delivery System for diagnostic, formative and summative testing, supporting the ACARA National Assessment Program including NAPLAN, for SCSEEC agreement prior to delivery (Element 3).

The key goals of the third element are:

a) to develop the capacity to deliver national assessments through an online environment, including the delivery of faster, richer assessment results to teachers, schools and families/carers; and

b) to achieve substantial efficiency savings through replacement of paper‐and‐pen assessments with online delivery.

ACARA has developed a comprehensive research agenda that will provide findings on a range of issues and provide evidence that will allow decisions to be made regarding the transition of NAPLAN from a pen‐and‐paper test to a computer‐based assessment. In 2014, ACARA will conduct a development study to finalise the targeting and measurement aspects of the tailored test design and its branching mechanism.

1.2 This Document

The purpose of this document is to articulate the ACARA high‐level business requirements to support the NASOP 2014 Development Study.

4

National Assessment and Surveys Online Program (NASOP) Development Study 2014 Appendix 1 Business Requirements – Testlet and Test Creation, Review and Delivery System,

2. Business Processes This section is used to provide a summary of the key conceptual stages.

2.1 Listing

The following listing provides a summary of the business processes required for the delivery of the 2014 development study. Each stage is further detailed in section 3 – Business Requirements.

Business Process Reference

Business Process Title Applicable Requirements

Test Development (TD)

Test item ingestion and testlet, fixed test and main test construction and review.

TD1 Ingest Test item TD1‐1 to TD1‐15

TD2 Construct test TD2‐1 to TD3‐10

Pre‐test Administration (PTA)

Preparing for the testing at school level

PTA2 Readiness for test at the local level PTA2‐1

PTA3 Register for test PTA3‐1 to PTA3‐15

Test Administration (TA)

The administration process employed during the testing cycle

TA1 Deliver tests TA1‐1 to TA1‐38

Marking and Scoring (MS)

The marking and scoring of the test responses, including release of the marked and scored data to analysis and reporting.

MS1 Mark and score tests MS1‐1 to MS1‐7


3. Business Requirements

This section provides further information on the business processes contained within section 2.2 Listing.

3.1 NASOP 2014 DEVELOPMENT STUDY ‐ STAGES

The following generic business requirements have been identified in relation to the NASOP 2014 Development Study conceptual stages as per 2.1 – The Model.

3.1.1 Identity

The online solution will:

1. provide OR have access to an identity management function.

2. support ACARA managing service provider and reviewer user accounts and authorisations.

3. accurately link the test item to the test response; the student to the test item response; the student to the class; the student to the school; the class to the school; the school to the jurisdiction; the school to responsible authority.

3.1.2 Security


1. provide a high‐level of security to stop unauthorised access.

2. securely store all NAPLAN related information.

3.1.3 Standards


1. adhere to the Web content Accessibility Guidelines version 2.0 (WCAG 2.0) and include any functionality required to enable compliance with the Disability Standards for Education 2005, and achievement of the Adjustments for Students with a disability outlined in the National Protocols for Test Administration 2013.

2. adhere to the IMS Global (QTI 2.1 and APIP 1.0) standards and deliver all interactions specified in QTI 2.1, with agreed functionality.

3. be device‐independent and support all common access methods and browsers.

3.1.4 Auditing and Editing

1. The online solution will capture relevant audit data for all active sessions for reporting.

2. Every action within the online solution must be fully tracked, reversible and editable.

6


3.2 TD1 – INGEST TEST ITEM

This business process captures the activities required to ingest and review test items across the NAPLAN domains. Please note that it is not intended that items will be authored directly into the online solution. Items will be authored and finalised in a separate Item Authoring Tool. Items will then be transferred to this online solution for testlet and test construction and test delivery.

3.2.1 Business Requirements


Business Requirements

ID TD1

Name Ingest Test Item

Actors ACARA – Test Managers

ACARA – Psychometricians

Service Providers – Test Item Author

Expert Reviewers

External Experts

Preconditions 1. The actors are correctly set up within the online solution (Authoring).

2. The actors have received their unique sign‐on to the online solution (test development).

3. The standards for test item creation are developed and available to service providers.

4. The national assessment framework has been developed, implemented and understood.

Post Conditions 1. Actors have performed all their authorised activities within the TD1 – Test Item Ingesting business process.

2. Test items are available for TD2 – Trialling.

Assumptions 1. The online solution (authoring) will support all of the following service scenarios:

a. items that will be authored to specification outside the online solution, but will be uploaded into the online solution

b. items that will be purchased that don't need modification which will be uploaded into the online solution

Assumptions 2. That the online solution (authoring) will verify the identity and authorisations for all actors.

3. That test items will be sourced from multiple channels ‐ local and international ‐ and that each channel has its challenges.

4. That the process for item creation and review will be undertaken prior to any items being uploaded into the online solution.

5. That no item editing or modification will be conducted in the online solution.

6. That the majority of test items are written to allow automated machine marking.

7. That those items which are stored within ACARA’s Online Test Item Creation and Review System (OTICRS) and are compatible with the applicable standards can be migrated into the online solution (authoring).

8. That ACARA continues to have the final review and outcome decision.

9. That all test items include all relevant metadata, stem, stimuli, rubric, prompt, marking information, assets and IP permissions required.

10. That Intellectual Property (IPENDO) permissions information will not be captured within the online solution.

Requirement ID Requirement Description Priority

TD1‐1 The online solution (ingestion) will simultaneously handle test item uploads during periods of high demand and concurrent usage.

M

TD1‐2 The online solution (ingestion) will have sufficient server load capacity to deal with multiple users and a growing test item bank of content.

M

TD1‐3 The online solution (ingestion) will allow authorised users to add and retire a test item and/or sets of test items.

M

8


TD1‐4 The online solution (ingestion) will enable users to upload individual items and/or groups of items, including required metadata, across the NAPLAN domains.

M

TD1‐5 The online solution (ingestion) will support the selection and uploading of multiple test items.

M

TD1‐6 The online solution (ingestion) will capture the required metadata with the test items – see Appendix A for detailed listing.

M

TD1‐7 The online solution (ingestion) will enable authorised users to terminate an original test item and ingest a new test item.

M

TD1‐8 The online solution (ingestion) will enable authorised users to search and/or browse for, sort and filter by any metadata field.

M

TD1‐9 The online solution (ingestion) will enable authorised users to search and/or browse for multiple test items and/or sets of test items.

M

TD1‐10 The online solution (ingestion) will enable authorised users to search across multiple criteria.

M

TD1‐11 The online solution (ingestion) will ingest test item file attachments and associate them to the correct test item throughout the process flows.

M

TD1‐12 The online solution (ingestion) will provide multiple methods for uploading test items such as ftp.

M

TD1‐13 The online solution (ingestion) will support the selection and uploading of multiple test items from authorised service providers.

M

TD1‐14 The online solution (ingestion) will enable the user to customise their view, i.e., ability to hide and unhide columns, etc.

M

TD1‐15 The online solution (test development) will apply a unique identifier to a test item at the point of ingestion.

M


3.3 TD3 – CONSTRUCT TEST

This business process includes those activities required for the construction of testlets (used within tailored, multi‐stage test design of the NAPLAN Reading and Numeracy tests) and fixed tests (those not included within the tailored, multi‐stage test design) used within the main NAPLAN test.


Business Requirements

ID TD3

Name Construct Main Tests


ACARA ‐ Psychometricians

Expert Reviewers

Online solution – Test Development

Preconditions 1. The actors are correctly set up within the online solution (test development).

2. The actors have received their unique sign‐on to the online solution (test development).

3. The psychometric information from trialling has been captured against the test items.

4. The NAPLAN test templates are available within the online solution (test development).

5. The ingested test items are available for use.

Post Conditions 1. Actors have performed all their authorised activities within the TD3 – Test Construction business process.

2. The NAPLAN tests are ready for delivery.

Assumptions 1. That the online solution (test delivery) will be able to simultaneously administer differing versions of the NAPLAN tests (up to 10 as per the tailored, multi‐staged test design) to students for a given domain and year level.

2. That ACARA will continue to have the final review and outcome decision.


TD3‐1 The online solution (test development) will support the creation and management of multiple tests across all stages of test development process.

M

TD3‐2 The online solution (test development) will support the tailored, multi‐staged test design rules.

M

TD3‐3 The online solution (test development) will enable a testlet to be shared between different tests and/or test branches without duplication.

M

TD3‐4 The online solution (test development) will enable a test item to be included in multiple testlets and test forms without duplication.

M

TD3‐5 The online solution (test development) will use test templates during the construction of testlets, fixed tests and the NASOP Development Study test.

M

TD3‐6 The online solution (test development) will support multiple test versions. M

TD3‐7 The online solution (test development) will support the assignment of test items to testlets and fixed tests.

M

TD3‐8 The online solution (test development) will make the approved NAPLAN tests available for test delivery.

M

TD3‐9 The online solution (test development) will enable the review and approval activities for testlets, fixed tests and NAPLAN tests.

M

TD3‐10 The online solution (authoring) will enable the capture of timeframe parameters with the NAPLAN main test – see Appendix C for detailed listing.

M

10


3.4 PTA2 – READINESS FOR TEST AT THE LOCAL LEVEL

The activities carried out by the responsible authority ‐ school, diocese, education department or other authorised organisations or aggregations ‐ to ready their organisation, staff and students for the NASOP 2014 Development Study.


Business Requirement

ID PTA2

Name Readiness for test at the local level

Actors ACARA – Policy and Reporting

School – Principal (or delegate)

School – Teachers

School – Non‐Teaching Staff

School – Participating Students

Jurisdiction – Responsible Authorities

Test Administrator Contractor

Preconditions 1. The national NAPLAN procedures and guidelines are available for consumption by the jurisdictions, school authorities and schools.

Post Conditions 1. Schools – their staff, students and systems – are ready for the NASOP 2014 Development Study.

2. The school’s NASOP 2014 Development Study registration information (student) is complete and accurate within the school’s system.

Assumptions 1. That ACARA continues to specify readiness criteria.

2. The jurisdictions will continue to localise and implement the national procedures and guidelines for their local context.

3. There are criteria defined and available to enable schools to evaluate their readiness for the NASOP 2014 Development Study, including ICT infrastructure.

4. The participating school staff has sufficient computer access, knowledge and experience to assist with the administration of the NASOP 2014 Development Study.

5. The participating students have sufficient computer access, knowledge and experience to undertake the NASOP 2014 Development Study tests.

6. The participating students and NAPLAN assigned staff have had access to the online solution to familiarise themselves within standard functions.


PTA2‐1 The online solution supports the school’s activities by providing:

1. a means for students and staff to familiarise themselves with basic operations in the weeks prior to the NASOP 2014 Development Study beginning,

2. a means for checking and confirming connectivity,

3. a means for schools to access the pre‐test administration information, i.e., URLs to websites and pdfs.

M


3.5 PTA3 – REGISTER FOR TEST

The responsible authority ‐ testing agency, education department, diocese, school or other authorised organisations or aggregations ‐ updates participating school details, assigns staff to testing roles, registers participating students and affirms, based on outcomes from PTA2, school readiness to participate in the NAPLAN testing. The required test registration data is supplied by all participating schools to enable their participating students to undertake NASOP 2014 Development Study testing and for their school to be able to administer the NASOP 2014 Development Study.


12



ID PTA3

Name Register for test

Actors Jurisdiction – Responsible Authorities

School – Principal (or delegate)

School – NASOP 2014 Development Study Co‐ordinator

ACARA – National Incident Team


Preconditions 1. The actors have successfully completed all required PTA2 activities.

2. That the online solution (registration) has access to an authoritative list of participating schools.

Post Conditions 1. The school – staff and students – are fully registered to participate in the NASOP 2014 Development Study.

2. The pre‐test participation information for a student is captured and available to the online solution (test delivery).

Assumptions 1. The school is the authoritative source for student registration information.

2. An authoritative source for school registration information will be accessible to the online solution (registration).

3. The online solution (registration) will assure the identity of registered students.

Assumptions, continued

4. That the responsible authority may want to continue to gather their schools’ registration information and load it into the online solution (registration) on their behalf.

5. That the responsible authority will continue to check all loaded data and be able to update, add new and delete student, school and staff data as appropriate.

6. That the responsible authority will continue to want to assign NAPLAN testing roles to staff as part of ongoing test management operations.

7. That the responsible authority will require principals to confirm registrations prior to committing the updates to the online solution (registration).

8. That a help desk will be available to the schools during the NASOP 2014 Development Study testing period.

9. That the online solution (registration) will use the participating schools information in a number of business processes, including PTA2, PTA3, TA1 and reporting.


PTA3‐1 The online solution (registration) will offer multiple options for the loading of the school's registration information.

M

PTA3‐2 The online solution (registration) will ensure the security of the school's student registration information.

M

PTA3‐3 The online solution (registration) will validate the login of all users trying to access the online solution.

M

PTA3‐4 The online solution (registration) will capture audit data on the actions and changes the user makes during an open session.

M

PTA3‐5 The online solution (registration) will make the audit data available for reporting. M

PTA3‐6 The online solution (registration) will provide registered staff allocated to a NAPLAN testing role their own unique login.

M

PTA3‐7 The online solution (registration) will follow industry standards for the logout function.

M

PTA3‐8 The online solution (registration) will support the generation of at least 20,000 student unique logins for the NAPLAN 2014 Development Study tests.

M

PTA3‐9 The online solution (registration) will support the set‐up of the registration period.

M


PTA3‐10 The online solution (registration) will enable the school, and/or responsible authority to load their NASOP 2014 Development Study registration information.

M

PTA3‐11 The online solution (registration will provide an ability to capture accessibility (disability) adjustments for individual participating students.

M

PTA3‐12 The online solution (registration) will provide an ability to capture participation information for individual students.

M

PTA3‐13 The online solution (registration) will make the accessibility and participation information available for test administration, psychometric analysis and reporting.

M

PTA3‐14 The online solution (registration) will support the set‐up of the NASOP 2014 Development Study testing period.

M

PTA3‐15 The online solution (registration) will enable authorised users to search and/or browse across student, school and staff registration information.

M

14


3.6 TA1 – DELIVER TEST

This process supports the local delivery of the NASOP 2014 Development Study tests to participating students.

16



ID TA1

Name Deliver test

Actors School – Principals

School – NASOP 2014 Development Study Co‐ordinator

School – Participating Students

Jurisdiction – Responsible Authorities

ACARA – Test Managers



Preconditions 1. The pre‐administration activities have been completed and the school is ready to administer the NAPLAN testing.

2. The NAPLAN tests have been approved and are ready for delivery.

3. The online solution (CAT – tailored design engine) has access to the required tailored, multi‐staged test design rules.

4. The online solution (test delivery) has access to the online solutions (AI – tailored design engine) and the (marking and scoring).

Post Conditions

1. Students have had an opportunity to participate in NAPLAN tests.

2. Test delivery information is available for reporting.

3. Those student responses that require human marking are available to the expert markers.

4. Automatically scored information is available for reporting.

Assumptions 1. That the online solution (test administration) does not allow printing of the NAPLAN tests and/or individual test items.

2. That the test administrator will ensure venue security and accessibility to the online solution (test delivery).

3. That the test administrator is still required to do an 'on‐the‐day' physical check of the testing room.

4. That the test administrator will still be required to actively supervise during the test.

5. That the test administrator will ensure all relevant rules of behaviour and test instructions are understood by students prior to commencing the NAPLAN test.

6. That any walk‐in, additional students are not subject to a parental withdrawal or exemption from NAPLAN.

7. That the school‐based exceptions and anomalies are noted and can be evaluated against national procedures and guidelines.

8. That the investigating authority will have experience in evidence collection against national standards.

9. That ACARA will be able to capture the various sampling designs for the selection of samples (e.g., equating and calibration) as and when required.

10. That the online solution (registration) will contain the student information required to administer the NAPLAN tests to an individual student as well as the descriptive and/or demographic information about students required for psychometric analysis and reporting.

11. That the online solution (test delivery) will present a student with the correct NAPLAN testlet as per the tailored, multi‐staged test design rules.

Requirement ID

Requirement Description Priority

TA1‐1 The online solution (test delivery) will provide the means for the test administrator to monitor the students' progress through the NASOP 2014 Development Study tests.

M

TA1‐2 The online solution (test delivery) will provide the means for the test administrator to reassign or assign a student to a test group.

M


TA1‐3 The online solution (test delivery) enables the test administrator to register additional students on the day of the test.

M

TA1‐4 The online solution (test delivery) will allow for the administration of a specific test to a single student or group of students within a class or school.

M

TA1‐5 The online solution (test delivery) enables the reset of student passwords on the day of the NASOP 2014 Development Study test.

M

TA1‐6 The online solution (test delivery) will display key NAPLAN test instructions at appropriate stages of the test period.

M

TA1‐7 The online solution (test delivery) will provide the test administrator with the means that enables him/her to provide NASOP 2014 Development Study test access to additional (unexpected) students arriving on the test day.

M

TA1‐8 The online solution (test delivery) enables practice questions at the beginning of the NASOP 2014 Development Study tests.

M

TA1‐9 The online solution (test delivery) will enable a student to begin the NASOP 2014 Development Study test without undertaking any practice questions.

M

TA1‐10 The online solution (test delivery) provides the means for the test administrator to log any exceptions and anomalies, for a student and/or group of students.

M

TA1‐11 The online solution (test delivery) will provide required tools to support students during testing, i.e., If calculate use = yes for a test item, then the online solution (test delivery) will provide the student with an online calculator. If the test item requires the use of the ruler or protractor, then the online solution (test delivery) will provide the student with an online version for their use.

M

TA1‐12 The online solution (test delivery) will enable the test administrator to access accessibility options required and ensure such options are implemented for participating students.

M

TA1‐13 The online solution (test delivery) will capture accessibility options used by participating students.

M

TA1‐14 The online solution (test delivery) will capture audit data about the delivery of the NASOP 2014 Development Study tests, including: test start and end times.

M

TA1‐15 The online solution (test delivery) provides a means for the test administrator to suspend a test for an individual student and/or group of students.

M

TA1‐16 The online solution (test delivery) provides a means for the test administrator to monitor the number of student test responses submitted and escalate any anomalies as per national procedures and guidelines, i.e., 30 students participated in the test, 30 tests were recorded as submitted. If fewer than 30 tests submitted, then an appropriate participation code is recorded.

M

TA1‐17 The online solution (test delivery) captures information about student participation that can be entered by the test administrator, then audited, verified and updated by the responsible authority.

M

TA1‐18 The online solution (test delivery) will provide the means for the test administrator to commence the test delivery to the participating NASOP 2014 Development Study students.

M

TA1‐19 The online solution (test delivery) displays test items to the participating students enabling them to record responses.

M

TA1‐20 The online solution (test delivery) will provide the means for both the test administrator and student to monitor and view progress through the NAPLAN main test, i.e., test items remaining; time left to complete assessment, number of test items answered, etc.

M

TA1‐21 The online solution (test delivery) will enable the responsible authority to monitor test submissions for all the schools within their authority.

M

18


TA1‐22 Where a disruption occurs, the online solution (test delivery) will:

1. display a service disruption message to students and the test administrator

2. capture information about the disruption in an incident log that can be commented on by the test administrator

3. make the incident log information available for reporting.

M

TA1‐23 The online solution (test delivery) will display a service resumption message to students and test administrators upon the services being restored and capture resumption information within the incident log.

M

TA1‐24 Where service has been interrupted, the online solution (test delivery) will resume the test from the point the student’s test was interrupted.

M

TA1‐25 Where service has been interrupted, the online solution (test delivery) will apply, at the least, the unused portion of the total time allowed.

M

TA1‐26 The online solution (test delivery) will have the capacity to support all of the (potential 15,000‐20,000) participating students taking the NASOP 2014 Development Study on the same day and at the same time.

M

TA1‐27 The online solution (test delivery) will apply the tailored, multi‐staged test design rules required to ensure the student is provided with the correct test items.

M

TA1‐28 The online solution (test delivery) will enable participating students to confirm their submission of a completed test.

M

TA1.29 The online solution (test delivery) will capture all test items presented to each participating student – regardless of whether the test item was answered.

M

TA1‐30 The online solution (test delivery) will enable a student to save their responses to complete the NASOP 2014 Development Study test as per the national protocols.

M

TA1‐31 The online solution (test delivery) will enable a student to submit blank and/or partially completed test item responses.

M

TA1‐32 The online solution (test delivery) will enable a student to submit blank and/or partially completed tests.

M

TA1‐33 The online solution (test delivery) will enable a student to submit their NAPLAN test responses where the time allowance has expired prior to the student completing the test.

M

TA1‐34 The online solution (test delivery) will prevent a student from accessing other applications, URLs and/or resources not associated with undertaking the NAPLAN tests during the period of test delivery, including screen capture.

M

TA1‐35 The online solution (test delivery) will enable a student to review and change their responses prior to the test being submitted.

M

TA1‐36 The online solution (test delivery) will simultaneously administer differing versions of the NASOP 2014 Development Study test (up to 10 as per the tailored, multi‐staged test design) to students for a given domain and year level.

M

TA1‐37 The online solution (test delivery) will enable authorised ACARA users to enter sampling designs for its usage.

M

TA1‐38 The online solution (test delivery) will apply sampling designs to randomly select students for sample testing.

M


3.7 MS1 – MARK and SCORE TESTS

This process marks and scores the tests ‐ automatically scored (AS) by the online solution (marking and scoring). Quality assured (QA) manual marking of open constructed response items will be conducted separate to the online solution.

There are three categories of test items to be marked and scored:

1. Open constructed response items – will be manually marked outside of the online solution. 2. Multiple‐choice(s) response items – will be auto‐scored 3. Closed constructed response items – will be auto‐scored

Further item types requiring autoscoring may be added by ACARA. Item types requiring support from the online solution will be discussed during finalisation of the business requirements.

Automatic scoring is required for tailored, multi‐staged test delivery as the student’s response to a testlet determines which testlet is presented next during the test. Manually marked items cannot be marked until a student has submitted the test and the response data is extracted from the online solution.

20




ID MS1

Name Mark and score tests



Expert Markers

Online Solution (Marking and Scoring)

Online solution (AI – Tailored Design Engine)

Preconditions 1. The NASOP 2014 Development Study tests are being administered.

2. Participating students are submitting responses to the NASOP 2014 Development Study tests.

3. The online solution (marking and scoring) has access to the required marking and scoring information related to each test items and/or testlet.

4. The online solution (CAT – tailored design engine) has access to the required tailored, multi‐staged test design rules.

Post Conditions 1. Participating student test submissions are correctly marked and scored.

2. The raw marked and scored data is available for analysis and reporting

Assumptions 1. That where automated marking (auto‐scoring) is employed, quality review will be undertaken by qualified markers during the preparation for and training of the automated scoring system/s.

2. That escalation criterion is available to be defined within the online solution (marking and scoring).

3. That auto‐scoring of relevant test responses will be implemented within the online solution (marking and scoring).

4. That the online solution (marking and scoring) applies the specified tailored, multi‐stage test design rules.


MS1‐1 The online solution (marking and scoring) will access relevant marking and scoring information for a test item directly from the online solution (test item bank).

M

MS1‐2 The online solution (authoring) will contain the marking information to the test item within online solution (test item bank).

M

MS1‐3 The online solution (marking and scoring) will support automated scoring for multiple‐choice response items and closed constructed response items.

M

MS1‐4 The online solution (marking and scoring) will support online quality assurance activities.

M

MS1‐5 The online solution (marking and scoring) will support the application of the specified test design rules.

M

MS1‐6 The online solution (marking and scoring) will support the test design rules required to ensure the student is provided with the correct test items.

M

MS1‐7 The online solution (marking and scoring) will make all scoring, marking, quality and audit information available for reporting.

M


3.8 Non‐Functional Requirements

The following non‐functional requirements have been identified:

Detail Requirement Business Process

Performance << Software/human interaction – no of terminals to be supported, no of simultaneous users, geographical location, disk storage required if known.

Response times ‐ Response times for the user to the system will be sub second.

Traffic/Volume >>

Usability << User interface standards, navigational methods, on‐line help etc. Example;

The system must meet Web Content Accessibility Guidelines version 2.0 (WCAG 2.0) at level AA standard of the international web content accessibility guidelines from the WWW Consortium.

The system provides an easy‐to‐use interface for the user to run any report defined as being available for that user.

The system must be usable from such mobile devices as iPads and tablets.

The system will offer sufficient flexibility to enable increased usability based on the user’s selections or preferences.

It is easy for any user to build ad‐hoc queries based on the data elements defined in the system. >>

Service Standards

<< Detail requirements such as required hours of system availability, Support/Help Desk availability during working hours, etc. >>

Security << Describe requirements for management of users, setup of access, maintenance of passwords, etc. check company’s ICT Security Policy>>

Audit and Control

<< Detail requirements for audit including level of auditing and reporting of audit records. >>

Legal or Corporate

<< Are they any legal or corporate implications? What legislation must be followed? is there a need for confidentiality? Do privacy requirements need to be met. >>

Data << Data conversions/loads, sharing data with other systems? >>

Business Continuity/ Disaster Recovery

<< Detail business continuity or disaster recovery requirements. Standard ITD support is as follows;

Item Requirement

Business Hours 0800 to 1800?

First Response 1 hour?

Platform High or low availability?

Hardware Vendor Next business day?

Recovery Time Objectives

Software: 4‐hours recovery?

Data: 4‐hours recovery?

Hardware: next business day, plus recovery?

Data Centre: best efforts recovery (weeks)?

Architecture << Future scalability requirements, distributed vs centralised approach, requirements to interface with other data sources. >>

Privacy << Include responses to Privacy Toolkit. Describe privacy and confidentiality requirements. >>

22


4. Supporting documents

4.1 METADATA LISTING

The metadata listing will be provided to all organisations to register interest in this Request for Tender (RFT).

assessment and surveys (nasop)...national assessment and surveys online program (nasop) development...

Documents