3rd nrc meeting, 9-12 june 2008, windsor iccs 2009 main survey field operations

3rd NRC Meeting, 9-12 June 2008, Windsor

ICCS 2009 Main Survey

Field Operations

2 3rd NRC Meeting, 9-12 June 2008, Windsor

Field Operations - Overview

National Context Questionnaire

Main Survey Operations Overview

On-line Data Collection

Support Materials

Main Survey Timeline

Next Steps


National Context Questionnaire

Stage 2 of National Contexts Survey will start in March 2009 and will continue until October/November 2009

Survey will be conducted on-line (similar to stage 1)

NRCs will only need to update information from stage 1

NOTE: There are separate logins (username and password) for the sources available on-line:

– ICCS study homepage

– WebFTP server (2-stage login)

– National Context Questionnaire

– Survey Activities Questionnaire


Main Survey Operations Overview


InstrumentPreparation

Sampling and Preparing

Survey Administration

Data Entryand Verification

SURVEY

Codebook Adaptation

Rece

ivin

g M

ate

rial

Data

Sen

dout

Scoring






SURVEY

Codebook Adaptation

Rece

ivin

g M

ate

rial

Data

Sen

dout

Scoring


Verification ofAdaptations

TranslationVerification

Adapt Instruments & DocumentAdaptations

CorrectAdaptations &

TranlsateInstruments

CorrectInstruments &

AssembleBooklets

LayoutVerification

CorrectInstruments

(if necessary)

Printing &Send Out


NRCISC/IEA Sec./DPCtime

Time planning before administration

Layout verification

Implement changes &assemble booklets

Translation verification

Adapt instruments

2 weeks

1 week

1 week

3 weeks

4 weeks

Start data collection16

weeks

Instruments send out

Printing & send out

Translate instruments

Verification of adaptations

1 week

4 weeks


due dateNRC/ISC/IEAtime

Time planning before administration - SH

Layout verification

Implement changes &assemble instruments

Translation verification

Adapt & translate instruments

1 week

1 week

2 weeks

1 week

Instruments send out July 11, 2008

August 22, 2008

September 19, 2008

September 26, 2008

October 10, 2008

October 31, 2008Start data collection

Printing & send out3 weeks October 31, 2008

Adapt instruments

Verification of adaptations July 25, 2008

July 18, 2008

4 weeks

4 weeks


Meanwhile…






SURVEY

Codebook Adaptation

Rece

ivin

g M

ate

rial

Data

Sen

dout

Scoring


Contact Schoolsand get

within SchoolInformation

Perform Within-School

Sampling

Prepare List of Schools

and send it to DPC

Recruit and Train Test Administrators

DPC provides list of sampled

Schools

Recruit and Train Scorers

Translate School Coordinator Manual

Translate Test Administrator Manual


School Contact

If not yet done, send sampling frame (list of schools) to the IEA DPC

Sampling of schools has been or will be done by the IEA DPC

List of sampled schools has been or will be provided by the IEA DPC

Contact schools, ensure participation

Identify liaison person (School Coordinator)

Provide School Coordinator Manual

Record status for each sampled school in WinW3S

Adaptation and translation of School Coordinator and Test Administrator Manual (no verification on international level, modifications to some extend possible, also combining the two to only one document)


Within-School Sampling (Students)

Two levels: students and teachers

Identification of classes on Class Listing Forms

– Class name

– Number of students

Random within-school sample of classes

Identification of students within sampled classes on Student Listing Forms

Send list of sampled students to School Coordinator on Student Tracking Form

WinW3S serves as survey administration tool


Within-School Sampling (Teachers)

Identification of eligible target grade teachers on Teacher Listing Forms

Within-school random sample of target grade teachers

Send Teacher Tracking Form to School Coordinator


Within-School Sampling

WinW3S will support NRCs with

– Storing school contact information

– Listing of classes / teachers / students

– Sampling of classes / teachers

– Assignment of booklets

– Survey administration

– Tracking of participation

Will be subject to Data Management Seminar on July 8-11, 2008 in Hamburg


Assigning Questionnaires

School Questionnaire

Teacher– Every teacher will receive one questionnaire

Student– Achievement Booklet (one of seven rotated forms)– Questionnaire (one form, no rotated forms in main survey)– Regional Module Questionnaire

WinW3S will assign forms and provide labels for questionnaires


NRCSchools

Time Planning before Administration

time

Contacts schools

Decision to participate

Sends listing forms

List students/teachers

Samples teachers

Sends tracking forms

Sends questionnaires

Start data collection

Name School Coordinator

Samples classes

List classes/providenumber of teachers


School ContactInstrumentstime

Time Planning before Administration

Contact schools

Send listing forms

Sample teachers

Send tracking forms

Send questionnaires

Implement changes & assemble

booklets


Printing

Data collection

Sample classes






SURVEY

Codebook Adaptation

Rece

ivin

g M

ate

rial

Data

Sen

dout

Scoring


Monitor Test Administrators

Ad hoc Support

Arrange for Material Receipt

Train Scorers

Train Data Entry Staff






SURVEY

Codebook Adaptation

Rece

ivin

g M

ate

rial

Data

Sen

dout

Scoring


Organize Material for (direct) Data Entry

Organize Material for Scoring and Reliability Scoring

Check if Information and Material is complete

Get back to School

if necessary

Enter Participation Information






SURVEY

Codebook Adaptation

Rece

ivin

g M

ate

rial

Data

Sen

dout

Scoring


Adapt InternationalCodebooks to

National Questionnaires

Use NationalAdaptation Forms

Use adaptedCodebooks for

Data Entry in WinDEM






SURVEY

Codebook Adaptation

Rece

ivin

g M

ate

rial

Data

Sen

dout

Scoring


Conduct and Monitor Scoring

Conduct and Monitor

Reliability Scoring

Organize Material forData Entry


Scoring

All open ended items have to be scored according to scoring guide

Reliability scoring for open ended items for ~700 booklets (100 booklets per rotated form), ideally at the beginning of the scoring process, so that measures can be undertaken to improve reliability of scoring

Occupations of parents/guardians have to be scored according to ISCO 88

Subject to Marker Training on July 7-8, 2008, in Hamburg






SURVEY

Codebook Adaptation

Rece

ivin

g M

ate

rial

Data

Sen

dout

Scoring


Complete Participation Information

Verify Data

Enter Data


Data Entry

Data entry software (WinDEM) and international codebooks for the MS are provided by the DPC on August 18, 2008

National staff should be trained on data entry, even if all or parts are being outsourced

Computer workstations need to be set up

National Centers enter and verify data from paper questionnaires






SURVEY

Codebook Adaptation

Rece

ivin

g M

ate

rial

Data

Sen

dout

Scoring


Set of Final Instruments

National Codebooks and Documentation (NAF v. IV)

VerifiedData sets

TrackingForms

Reliability Scoring Data






SURVEY

Codebook Adaptation

Rece

ivin

g M

ate

rial

Data

Sen

dout

Scoring


On-line Data Collection


On-line Data Collection (ODC)

Used for all questionnaires that National Centers should answer

– National Context Questionnaire

– Survey Activities Questionnaire

International Option for

– School Questionnaire

– Teacher Questionnaire

Analysis of split half design during field trial showed that the use of the on-line mode in general is feasible

All nine countries participating in ODC option during the FT received individual feedback

Only countries that tried out ODC in FT can use ODC option for MS


On-line Data Collection in MS

True mixed mode design

Goal:

– Reduce response burden by offering respondents preferred mode

– Save time on data entry and data processing

– Improve quality of data

Teachers and school principals should be assigned to the preferred mode

– Knowlegde of NRC about feasibility/preferences

– School coordinator could contactS school principal and teachers

Ensure fall-back to paper questionnaire


Codebook Adaptation


Sampling and PreparingAdministration

Data Entry and

Verification

Ad

min

istr

ati

on

Rece

ivin

g M

ate

rial

Data

Sen

doutCodebook Adaptation

Prepare ODC

Monit

or

OD

C


School contactInstrumentstime

Time planning before administration ODC

Contact schools

Sample students / teachers

Send tracking forms

Send questionnaires

Implement changes & assemble

booklets


Printing

Data collection

PrintingPrepareODC

Send quest./cover letters

Send listing forms

Sample classes


On-line Data Collection (ODC)

Based on concept of cultures, one setup per language and country

Participants receive “cover letter”

– Label with login information provided by WinW3S

– Culture specific URL

– Personal username and password

– Sent to school coordinator for distribution

Same administration period as paper questionnaires

NRCs can monitor completion status of on-line questionnaires

NEW: Status of on-line respondents can be exported to WinW3S

If necessary, NRCs instruct School Coordinators to remind teachers/school principals to complete the questionnaire

No data entry at National Center needed


Support Material


Support Material

Survey Operations Procedures Manual (MS)– Unit 1: School contact and material preparation– Unit 2: Within school sampling and test administration

School Coordinator ManualTest Administrator ManualNational Quality Control Monitor ManualWithin School Sampling Software (WinW3S)


Support Material

Survey Operations Procedures Manual (MS)– Unit 3: On-line data collection, scoring and data entry

IEA SurveySystem DesignerScoring GuideCodebooksWindows Data Entry Manager (WinDEM)

Distribution of all materials and software packages through secure section of study website at http://iccs.acer.edu.au


Main Survey Timeline


Main Survey Timeline - SH

- Receive verified final instruments- Start printing and distribution of questionnaires- Send final instruments and NAF v. IV to IEA DPC- Set up on-line questionnaires

October 10, 2008

Submit assembled instruments and NAF version III for layout verification to ACER

September 26, 2008

Receive verified instrumentSeptember 19, 2008

Submit translated questionnaires and NAF version II for international translation verification to IEA Sec.

August 22, 2008

Receive feedback on national adaptationsJuly 25, 2008

Submit national adaptations in NAF version I to ISCJuly 18, 2008

Receive instruments & NAFJuly 11, 2008


Main Survey Timeline - SH

Final deadline for submitting data to DPCMarch 27, 2009

- Scoring of open-ended items and occupations- Data entry and verification

November –March, 2009

Proposed MS administration period (subject to country conditions)

October – December, 2008

Submit on-line questionnaires definition files for verification to DPC

October 17, 2008

Give feedback about FT procedures using theon-line Survey Activities Questionnaire

April, 2009


Main Survey Timeline – NH

- Receive verified final instruments- Start printing and distribution of questionnaires- Send final instruments and NAF v. IV to IEA DPC- Set up on-line questionnaires

February, 2008

Submit assembled instruments and NAF version III for layout verification to ACER

February, 2008

Translation verificationJanuary, 2008

Translate questionnaires and prepare NAF version II for international translation verification at IEA Sec.

December, 2008

Receive feedback on national adaptationsNovember, 2008

Submit national adaptations in NAF version I to ISCNovember, 2008

Receive instruments & NAFJuly 11, 2008


Main Survey Timeline - NH

Final deadline for submitting data to DPCAugust 28, 2009

- Scoring of open-ended items and occupations- Data entry and verification

April –August, 2009

Proposed MS administration period (subject to country conditions)

March – May, 2009

Submit on-line questionnaires definition files for verification to DPC

February, 2008

Give feedback about FT procedures using theon-line Survey Activities Questionnaire

September, 2009


Next Steps


Next Steps

Prepare team at national center and plan for

– School contact

– Adaptation & translation of instruments

– Test administrators

– Scorers (CCE items and occupations)

– Data management (data entry and verification)

Be prepared to receive Main Survey instruments on July 11, 2008


Next Steps

Marker Training on July 7-8, 2008 in Hamburg

Data Management Seminar on July 8-11, 2008 in Hamburg


Contact Information


Contact Information

Field Operations / Data Management:

IEA Data Processing and Research Center

Falk Brese Michael Jung

[email protected]


Thank you for your attention!

3rd nrc meeting, 9-12 june 2008, windsor iccs 2009 main survey field operations

Documents