Dr. Debbie Hahs-VaughnDr. Mary E. Little

Advance Organizer

Changes in Teacher Education-325T OSEP Overview of 325 T and Evaluation teams UCF project and procedures Evaluation Questions, procedures, and

discussions BREAK Research Designs and Methods-Slavin

Quantitative Designs-Chapters 2-6 Qualitative and Action Research-Chapters 7-9 Planning and Implementing Research-Chapters 10-

12)

Think-Pair-Share

Consider the issues raised by Brownell et al and Ravitch (among others) re: teacher education and the need for change within teacher education and special education.

Name and defend several major goals for change.

Consider specific priorities for innovative projects in teacher preparation in special education within institutions of higher education. What priorities should be established by OSEP?

325T RFP Initiatives-OSEP

Conclusion of 325 competitions focused on personnel preparation in high incidence disabilities.

New competition, 325T, began in 2006. Focus of the priority: enhance and revise current teacher

preparation programs to address current needs in teacher preparation through more collaborative programming in the following domains: A-Licensure and State Certification (Credentials) B-Organizational Structure C-Curricular and Course Content D-Student Support (Induction and Mentoring) E-Program Evaluation

Annual competition of federal awardsClearly articulated components to addressMonthly updates and collaborative discussions across sites and among grant recipients to share research, procedures, and activities.Collaboration across stakeholders (colleges, departments, school district and state personnel) Program revision and evaluation; continued funding based upon approval of program enhancements for duration of five years, if continued.

325 T Program enhancement grants

Co-Principal Investigators: Mary Little, Ph.D., Suzanne Martin, Ph.D., Martha Stewart, Ph. D.

Program Evaluator: Debbie Hahs-Vaughn

Purpose of Project

To enhance existing, approved initial certification programs to prepare highly-qualified teachers of students with high incidence disabilities (HID) within the Masters program. The major content

focus will be math and science instruction for students with HID,

including CLD students, in co-taught and resource settings within urban middle school that have not met AYP targets.

Improve recruitment to increase the number of highly qualified teachers in ESE, especially from diverse backgrounds;

Enhance and implement an effective preparation program focusing on math and science content instruction at the middle school level within the ESE Masters and certification to assure HQT status and skills to successfully teach in urban settings;

Improve mentoring and induction;

Implement technology to support overall program effectively.

Our Task:

With our colleagues at UCF and using the resources provided in readings, websites, and in class, develop a program evaluation plan, instrumentation, and methodology to address one of the four goals form the Program Evaluation Plan of this OSEP 325-T project.

Specific resources: Assignment overview and rubric; Program evaluation overview; Summary for UCF 325T project; Team of doctoral students and professors.

Evaluation Team: Goal 1

Improve recruitment to increase the number of highly qualified teachers in ESE, especially from diverse backgrounds; Julie Joseph Jonathon Rasmussen Eric Wells Ian Gesundheit

Evaluation Team: Goal 2

Enhance and implement an effective preparation program focusing on math and science content instruction at the middle school level within the ESE Masters and certification to assure HQT status and skills to successfully teach in urban settings;

Kathy Becht Betsy Theis Jenn Gallup Erika Niecsinger

Evaluation Team: Goal 3

Improve mentoring and induction Stacy Hardin Jillian Gourwitz Hannah Ehrli Kelly Schaeffer

Evaluation Team: Goal 4

Improve and enhance technology to support overall program effectively. Barb Serianni Aaron Koch Onur Kocaoz Lauren Delisio

UCF Faculty, Bridges

Program Staff & UCF

Students

Middle/High schools-Districts

HQ/NBCT Teachers, Students,

CounselorsParents

State and Community Partners –

FLDOE, CEC, NMTE, NSTE,

Industry, Volunteers

Technology -Lab facilities & equipmentFLDOE-ATEN

USDOE Cooperative

Funding & IRIS, FLDOE and UCF

Resources

Assemble Stakeholders

& Solicit Input

Model Exceptional Education Graduate Program (multiple

pathways-content and

delivery)Curriculum modules

aligned with state/federal standards for

HQT

UCF ESE Students

prepared to use

technology in classroom

Technology: Recruitment, Instruction, Coaching/ Mentoring

(especially CLD)

Develop Virtual

Activities – Mentoring, Instruction,

Collaboration

Establish Bridge

Program Team

Review and Revise ESE Graduate

Curriculum – Integrate New Technologies

Recruit ESE Graduate

Students & Gather Baseline

Data

UCF ESE Students

prepared to teach ESE students in

Math/Science

Establish data collection system to

analyze/revise program

Dissemination&

Presentations

Increase in the number of teachers from

CLD backgrounds

Increase in the

technology for program communicati

on and improvement

Increase the knowledge and skills in math

and science of teachers-HQT

Increase in teachers’

understanding of

math/science concepts

Improved Mentoring and

Induction Activities

Increased knowledge of math/science

content

Effective use of technology

to support overall

program effectively

Increase in the number of teachers from CLD

backgrounds

Increase the retention of

highly qualified

teachers into schools

Improve recruitment to increase the number of highly qualified teachers in ESE, especially from diverse backgrounds;

Enhance and implement an effective preparation program focusing on math and science content instruction at the middle school level within the ESE Masters and certification to assure HQT status and skills to successfully teach in urban settings;

Improve mentoring and induction;

Implement technology to support overall program effectively.

EEX 7320Presented by: Dr. Debbie L. Hahs-Vaughn

Sections 1 & 2

Select ‘evaluation goal’ and ‘evaluation subgoal’

Section 3: Research Question

There are multiple types of research questions and resulting research designs that can be used to answer the subgoals

Section 4:Instrumentation

Most applicable to: Survey research Other research designs that employ a test or

assessments Elements listed in the procedures template may

not be applicable to and/or will need to be adapted for other types of instrumentation, such as: Content analysis (instrumentation should describe the

rubric/coding form used to gather data) Interviews (instrumentation should describe interview

protocol) Factual data collection (may have a “data source”

section other than instrumentation section)

Section 5:Description of Sample/Population

‘Who’ will participate (i.e., unit of analysis, could be people, such as faculty or students, or documents, such as syllabi)

How they will be selected (census if population or sampling method if sample)

How many anticipated (i.e., anticipated sample size)

Section 6:Data Collection Procedures Who (any process for selecting the sample that was

not transparently and comprehensively detailed in section 5 should be explained here. In addition to other “who’s” applicable to the data collection procedures)

What (what will be done to them or gathered to get the measured data)

Where (where will data be gathered e.g., online survey, in-class, interviews conducted at…and other where’s that are applicable to the data collection process)

When (anticipated timeline) How (how will activities be performed) KEY IS TRANSPARENCY AND COMPREHENSIVENESS

Section 7:Data Analysis

Data analysis should correspond to the research question Quantitative analysis

Descriptive Inferential

Qualitative analysis (see next slides)

Leech, N. L. & Onwuegbuzie, A. J.  (2008).  Qualita-tive data analysis: A compendium of techniques and a framework for selection for school psychology research and beyond.  School Psychology Quarterly, 23, (4), 587-604.

Reliability and Validity in Qualitative Data

Anfara, V. A., Brown, K. M, & Mangione, T. L. (2002). Qualitative analysis on stage: Making the research process more public. Educational Researcher, 28-38.

Kathy Becht, Lauren Delisio, Aaron Koch, Onur Kocaoz, and Erika Nicsinger

Chapter 10: Measurement

• Reliability o Reliability is the degree to which a measure is

consistent in producing the same readings when measuring the same things.

o Ex.: a rulero Low Measures of Reliabilityo Importance of Reliability

• Validity o Validity refers to the degree to which a measure

actually measures the concept it is supposed to measure.

o A measure can be reliable, but not valid. o Types of Validity

Chapter 10: Measurement

• Types of measureso Achievement and Aptitude Tests

Descriptions Ex.: IQ Tests, IOWA Tests, Stanford Achievement Test

o Standardized Tests Description Ex.: FCAT

o Criterion-Referenced Tests Description Ex.: driving test, classroom assessment

o Questionnaire Scales Description Ex.: CBAM, Likert-scaled questions

• Advantages and Disadvantages

Chapter 10: Measurement

•o Useo Constructiono Caution against bias

• Determining Adequate Sample Size

Behavioral Observation

Chapter 10: Measurement

• Connections to 325T Granto Valid and reliable measures are needed to evaluate the

efficacy of outcomes.o How will you know if the program has met the project goals?

• Additional Resources o Web Resources:

Web Center for Social Research Methods, Knowledge Methods Resource Base: http://www.socialresearchmethods.net

The Mental Measurements Yearbook: http://www.ebscohost.com/academic/mental-measurements-yearbook

Educational Testing Service: http://www.ets.org/tests_products

o Print Resources: Drost, E.A. (2011). Validity and reliability in social science research. Education Research and

Perspectives, 38(1), 105-123.

Gall, M. D., Gall., J. P., & Borg, W. R. (2006). Educational research: An introduction (8th ed.). Boston: MA: Allyn & Bacon.

Chapter 11: Threats to Internal/External Validity

•o Internal validity is the degree to which the result of a

study can be attributed to the treatments or other independent variables identified in the hypothesis.

• External Validityo External validity is the degree to which the findings of

a particular study have meaning for other settings and samples.

Internal Validity

Chapter 11: Threats to Internal/External Validity

• Threats to Internal Validity

o Threats: history, maturation, testing effects, instrumentation effects, ceiling effect, floor effect, selection bias, statistical regression, attrition, confounding variables.

o Examples of Internal Threats

• Threats to External Validity o Threats: lack of internal validity,

nonrepresentativeness, artificiality, reactivity. o Examples of External Threats

Chapter 11: Threats to Internal/External Validity

• Connections to 325T Granto Internal validity is the degree to which we eliminate confounding

variables inside the study which can cloud the results of our research and minimize the efficacy of our grant.

o External validity is the generalizability to larger audiences or settings which is needed to justify our grant outcomes.

• Additional Resources o Web Resources:

Meissner, H. & Vinson, C. (2011). External Validity: Why It Matters [PowerPoint slides]. Retrieved from http://conferences.thehillgroup.com/OBSSRinstitutes/TIDIRH2011/presentations /Aug5/Meissner&Vinson_TIDIRH%20External%20Validity%20Print%20Version.pdf.

o Print Resources: Gall, M. D., Gall, J. P., & Borg, W. R. (2006). Educational Research: An Introduction (8th ed.).

Pearson.

Steckler, A., & McLeroy, K. R. (2008). The Importance of External Validity. American Journal of Public Health, 98(1), 9–10. doi:10.2105/AJPH.2007.126847

Chapter 12: Planning and Implementing the Study

• Planning the Study

• Selecting the Problem

• Literature Review Components

SynthesisMethodologySignificance of Research ProblemWriting Style

Coverage

Chapter 12: Planning and Implementing the Study

• Writing a Proposal statement of the problem hypothesis brief literature review detailed procedures schedule roughing out and discussing proposal evaluation of the proposal IRB

Chapter 12: Planning and Implementing the Study

• Planning Study Procedures Manuals

Testing instructions

Ethical research

Chapter 12: Planning and Implementing the Study

• Gaining Access to Schools and Implementing the

Study FERPA Integrity Maintain group size & quality Monitor

Report to participants

Chapter 12: Planning and Implementing the Study

• Connections to 325T Grant:o Literature Review Format

http://ncipp.education.ufl.edu/files_31/LlitReviewCourseMatching.pdf

• Additional Resources:o Web Resources:

What Works Reporting Guide for Study Authors. http://ies.ed.gov/ncee/wwc/pdf/reference_resources/wwc_gsa_v1.pdf

o Print Resources: Slavin, R. E. (2007). Educational research in the age of

accountability.(p. 225). Boston, MA: Pearson. Boote, D., & Beille, P. (2005). Scholars before Researchers:

On the centrality of the dissertation literature review in research preparation. Educational Researcher: (34) 6 pp. 3-15.

Slavin Chapter 10-12

• What questions do you have regarding:o Measuremento Internal Validityo External Validityo Planning the Studyo Implementing the Studyo Connections to 325T Granto Additional Resources

Where are we with Teacher Quality?

http://education.ufl.edu/grants/ncipp

Our Course Questions:

What questions can be addressed by qualitative methodologies? (Slavin)

How do we use data to implement change? (ALL) What types of data should be collected and used for

different purposes? (Slavin…then, used and discussed) What is my role as the researcher? (Personal reflection) How do we collect and use data to make change?

(Collins; Guskey; some course readings/reflections) What different models of program evaluation are there

to use? (Research completed…now used/discussed) How do we begin this process? (325T and Ind. Projects) What data sources should be used? (Slavin; Guskey-

applied)

Next Steps:

Review all materials, websites provided Review chapters/resources-Slavin Complete ONE response-Wiki-Elmore Begin Program Evaluation of your

section of 325T authentic learning assignment according to directions and rubric

Prepare to meet with your team next week during class time after 1 hour presentation