1 session 3: framework for a coding guide october 27, 2009 2:00-3:30 pm (edt) instructor: chad nye

1

Session 3: Framework for a Coding Guide

October 27, 20092:00-3:30 PM (EDT)

Instructor: Chad Nye

2

Dr. Chad’s Wonder Drug

Life’s Elixir of Youth, Beauty, Wisdom & Painless Coding

3

A wholesome blend of natural extracts of thirty different herbs and root vegetables that promotes and revitalizes the capacity for learning and memory, strengthens the nerves, and effectively wards off cataleptic neuroplexy AND coding anxiety.

4

Inclusion and Exclusion Criteria

Reduce Reviewer Bias: Criteria are determined

before beginning Information Retrieval

‘Standardize’ the process and resulting decisions

5

Inclusion and Exclusion Criteria: Potential Categories

Study Design

Population

Intervention

Outcomes

Measures

Publication Sources

6

Coding for Fun and Profit

Dr. Chad’s Wonder Drug taken 2 times a day will result in the complete life experience of the life well lived!!!

7

Coding is a 2 Step Process

Stage 1 – Title and Abstract Eliminate studies that are clearly inappropriate Used a limited set of criteria

8

Inclusion CriteriaAbstract & Title Stage

Studies will meet at least one of the following criteria:

1. Participant The participant sample of study must include only adults 18

years or older, with a diagnosis of Autism Spectrum Disorder.

2. Intervention The focus of the study intervention must center on the topic of

employment.

9

Inclusion CriteriaAbstract & Title Stage

Studies can be advanced to

the next stage if:

1. They meet the criteria OR

2. If these criteria are not clear from the title or abstract

10

Coding is a 2 Step Process

Stage 1– Title and Abstract Eliminate studies that are clearly inappropriate Used a limited set of criteria

Stage 2– Full Text Identify the studies that will be included in the review itself Use full set of criteria

11

Inclusion CriteriaFull Text Stage

Research Design

Studies use an experimental (RCT), quasi-experimental design (QED), and single subject experimental design (SSED)

Participants

Individuals with an autism spectrum disorder who are 18 years of age or older and not enrolled in a secondary school program (e.g., high school)

12


a. Research Design

b. Participants

c. Intervention

The treatment must be directed towards the provision of vocationally oriented services intended to produce employment outcomes for individuals with autism spectrum disorders.

13


a. Research Design

b. Participants

c. Intervention

d. Outcome Measures

The study must report subsequent attainment of an employment placement and provide specific data about the duration and/or retention of that placement. Employment will be considered to encompass full or part-time (25% time or more) placements. Interventions of any length duration will be included provided adequate description is provided.

14


a. Research Design

b. Participants

c. Intervention

d. Outcome Measures

e. Publication Status

Eligible studies may be either published or unpublished, written in any language, and conducted in any country.

15

Inclusion/Exclusion Decision Tree

Potentially relevant reports identified

(n=583)

Reports progressing to next level

(n=60)

Reports used to code studies for inclusion

(n=20)

Reports excluded at Phase II screening

(n=40)

Reports excluded by title and abstract

review(n=523)

16

Study Coding*

Purpose of Coding1. Identify all those variables that you believe

might make difference in the observed results.

2. Organize those individual variables into categories of characteristics in order to evaluate the quality of the study

3. Try to determine what is really important about the study

4. Quantify both text and data for analysis

17

Coding Form5. Guide for locating and recording the who,

what, where, how, when, and how much of a study (aka-variables)

6. A record of the ‘interpretation’ of the variables that can be independently replicated (aka-a monitor on the accuracy of judgments)

7. Provides the basis for the data arrangement at the analysis stage

18

Study Coding

Categories of Study Characteristics

Design

Participant

Treatment

Outcome

19

Design Characteristics

Type of Design RCT QED Pre-Experimental Single Subject

Blinding Attrition Sampling Method

20

Participant Characteristics

Information is collected for BOTH Experimental AND Control groups.

Number of Participants Age Gender Severity Language Setting

21

Treatment Characteristics

Type of Tx Method Length of Tx Program Tx Mode: Group-Individual Number of Tx Sessions Length of Tx Sessions Dosage

22

Outcome Characteristics

Outcome Variable(s) Outcome Measure(s) Assessor of Outcome Assessor Qualifications Time Post Tx to Outcome Measure

Immediate Follow-Up

23

CodeBook A dictionary of all the terms used in the

Coding Form

A guide for all coders to follow to reduce error and bias

A reference source for accurate replication

24

A. Publication Source

1. Journal 2. Dissertation 3.Thesis

4. Unpublished Paper 5. Book 6. Other: ________

B. Subject Characteristics (page(s): ____________)Group Pretest Posttest Attrition Age Grade(s) Males (n) (n) (n) (n) (n) (n)

Tx: _22___ _22__ ______ ______ ______ _____ Cp: ______ _____ ______ ______ ______ _____

Co: _22___ __18_ __ 4___ ______ ______ _____

25

C. Socioeconomic Status (page(s):___________

1. Low 2. Middle 3. Upper

4. Low-Middle 5. Middle Upper 6. Labeled Mixed

7. Unclear 8. ___________ 9. NR

Comments:__________________________________

26

D. Sample Source (page(s)_________)

1. Public School

2. Secular Private School

3. Religious Private School

4. Other:_________________

5. NR

Comments

27

E. Study Setting (page(s) ______)

1. Urban 2. Suburban 3. Rural

Geographic Setting: ______________________

Comments:

28

F. Author’s Labeling of Participants’ Achievement Levels (page _____)

1. Average Achieving 2. High Achieving 3. Underachieving

4. At-Risk 5. Unclear

6. Other:___________________ 7. NR

G. Race/Ethnicity (page ______)

1. African American 2. American Indian 3. Asian

4. Hispanic/Latino 5. White

6. Other: ____________________ 7. NR

29

H. Intervention Characteristics

1. Type of intervention (page __________________)

2. Total length of intervention program: __________ or NR (page _____)

3. Length of time PER intervention session:________ or NR (page _____)

4. Number of sessions: ________ per ________ or NR (page _____)

5. Describe the intervention(s) intended for delivery to the study groups including how and when it was administered.______________________________________

30

6. Fidelity of implementation data reported? (page(s)______) 0=No 1=Yes

If yes, describe the intervention(s) intended for delivery to the study groups including how and when it was administered:

________________________________________________________________________________________________________________________________________________________________________________________________

7. Intervention implemented as described? (page(s) _____)

1=No 2=Yes 3= Somewhat 9=NR

31

I. Outcome Measure(s)

Type of Measure Instrument

Variable(s) Who Administered the Measurement Instrument?

1. Norm Referenced Test (page____)

___________________________

a. Parent b. Teacherc. Researcher d. Othere. NR

2. Criterion Referenced Test (page____)

__________________________


4. Rating/Survey (page____)

__________________


5. Class Grade (page____)

___________________________


32

J. Design Characteristics

Recruitment Pool (page_____)

1. Referral 2. Criterion 3. Norm Reference Test4. Existing Group 5. Cohort 6. Volunteer7. Other ___________________ 99. NR

Research Design Type (pg____)

1. Group (Cluster, Place-based) Randomized Design2. Individual Randomized Design3. Quasi-Experiment: Interrupted Time Series Design4. Quasi-Experiment: Regression Discontinuity5. Quasi-Experiment: Nonequivalent Comparison Group Design6. Single Subject Design7. Other Designs: _____________________________________

33

K. Assignment Procedures (If applicable, otherwise skip to section I)

Subject Assignment (page(s) _____)

1. Individual Random

2. Group Random

3. Individual Match Random

4. Quasi-random

34

2. Method of Random Allocation (page(s) _____)

Method of Random Allocation Who Implemented the Random Allocation?

1. Random Numbers (page____) a. Parent b. Teacherc. Researcher d. Othere. NR

2. Coin Flip (page____) a. Parent b. Teacherc. Researcher d. Othere. NR

4.Other:_______(page____) a. Parent b. Teacherc. Researcher d. Othere. NR

35

Was concealment of allocation used as part of the randomization procedures? 1=no 2= yes 3=NR

a. If yes, were the allocation procedures concealed from any of the following people?

1. Researcher (pg _____)

2. Participant (pg _____)

3. Intervener (pg _____)

4. Assessor (pg _____)

36

I. Quasi-Experimental Assignment Procedures (If applicable, otherwise skip to section J)

1. Was any of the following research design techniques used to equate groups prior to the intervention?

• a. Matching on pretest proxy (pg _____)• b. Matching on a pretest (pg _____)• c. Matching on other variables (pg _____)• d. No Matching (pg _____)

2. Was statistical equating used prior to the intervention? 1=no 2=yes 3.NR

Comments:________________________________________

37

J. Effect Size Characteristics

Comparisons: Group 1: _TX___vs. Group 2:_Ctl___

Outcomes _Reading___ __________ ___________ Groups __1__ _2_ __1__ __2__ __1_ __2__

Mean _22__ __18_ _____ _____ ____ _____

SD _3.55 _4.22_ _____ _____ ____ _____N at Posttest _____ _____ _____ _____ ____ _____

F value _____ _____ _____ _____ _____ ____Chi-square _____ _____ ___ _____ _____ _____ ____

p value _____ _____ _____ _____ _____ _____ t value _____ _____ _____ _____ _____ _____

38

Method of Analysis (pg _____)

a. Intention to Treat:

b. Treated Participants Only

39

Excerpt from Master Coding Guide

for Study Coding

-------------------------------------

Code the Fantuzzo Study

40

Data Extraction Practice

1 session 3: framework for a coding guide october 27, 2009 2:00-3:30 pm (edt) instructor: chad nye

Documents