msp project evaluation lessons learned margaret phelps & kenneth w. hunter, sr. january 8, 2008...

MSP PROJECTEVALUATION

Lessons Learned

Margaret Phelps & Kenneth W. Hunter, Sr.

January 8, 2008

Millard Oakley STEM CenterMillard Oakley STEM Center

Our MSP History

Upper Cumberland Middle Grades Math Partnership (2004 - 2006)

TTU Engineering (7-12) Math Science Partnership (2005 - 2007)

Upper Cumberland Middle Grades Math Partnership (2007 - 2009)

Upper Cumberland Middle Grades Science Research Partnership (2008 - 2010)

Tennessee Pre-Engineering (7-12) Math Science Research Partnership (2008 - 2010)

Formative Assessment

Participant Surveys (Institutes & Workshops)

– Content– Format– Faculty– Scheduling– Facilities & food

Basis for change/

improvement

Formative Assessment

Daily faculty meetings during summer institutes

– Observations of master teachers• That was great! The teachers loved it!• Don’t ever do that again!

– Open discussion of issues• Extremely helpful for “non-education” faculty• Basis for “quick fix” improvements

Evaluation Designs

Experimental Design with Random Assignment– LEAs provide list of eligible teachers (or teams)– Local evaluator does random selection

Non-matched Comparison Groups– LEAs select teachers for intervention and control

groups– Groups are compared statistically on base line

measures to determine equivalence

Descriptive--No comparison group

Evaluation Components

Teachers– STEM Content Knowledge– STEM Pedagogy & Beliefs– Classroom Performance

Students– STEM Content Knowledge– STEM Authentic Assessment

Teacher Content--Praxis II

Advantages– External measure– Valid & Reliable– Tracks change over

time

Disadvantages– Cost– Obtaining score

reports– Inflexibility of

schedule

Instructor-made Test

Advantages– Matches institute

content– Aligns with standards– Control of testing

Disadvantages– Not necessarily valid

or reliable– Not normed– Forms might not be

equivalent

Purchased Test

Advantages– Control of testing– Access to scores

Disadvantages– Cost– Single form– Alignment

Teacher Pedagogy & Beliefs

Teacher-developed lesson modules

Reports on lesson implementation

Periodic self-assessment

Surveys– Beliefs and Attitudes (Math)– Misconceptions (Science)

Classroom Performance

Project-specific Observations – Conducted by LEA liaison or principal– Informs local leadership– Creates buy-in– Not valid for research purposes

Research-based Observations– Requires training of observers– Cost (perhaps)– Useful for research purposes

Student Content Knowledge

Middle school math & science– State testing program– Password-protected data available

(can be disaggregated for analysis)

– Compare NCE and proficiency level of students for pervious year (pre-) to year with project teacher (post-)

Student Content Knowledge

High school math & science– 8th grade scores are only common pre-test– HS scores not reported in NCE– Proficiency levels do not provide basis for

strong evaluation– Many courses not tested– ACT only available for graduates

Student Content Knowledge

Engineering Principles– JETS Assess

(Junior Engineer Technical Society)– Will be administered in classrooms– Fall/Spring for comparison– Control of data collection– Will also be administered to teachers

Student Authentic AssessmentElectronic Portfolio

Based on Engineering Principles

Multi-year – leads to pre-engineering certificate

Curricular components– Project Lead the Way– Math, Science & Career Technical courses &

projects

Student Authentic AssessmentElectronic Portfolio

Co-curricular components– FIRST LEGO League, FIRST Robotics– 4-H and Scouting projects– Work experience– Service learning– Utilize local opportunities

Contact Information

Dr. Margaret Phelps – mphelps@tntech.edu– All TTU-related MSP programs & Millard Oakley STEM Center

Mr. Kenneth W. Hunter, Sr. – khunter@tntech.edu– Tennessee Pre-Engineering Math Science Research Partnership (2008-2010)

Dr. Susan Gore – sgore@tntech.edu– Upper Cumberland Middle Grades Science Research Partnership (2008-2010)

Dr. Sandra Koczwara – koczwaras@k12tn.net– Upper Cumberland Middle Grades Math Partnership (2004-2006)– Upper Cumberland Middle Grades Math Partnership (2007-2009)

Dr. Roy Loutzenheiser – rloutz@tntech.edu– TTU Engineering Math Science Partnership (2005-2007)

The instructional practices and assessments discussed or shown in these

presentations are not intended as an endorsement by the U.S. Department of

Education.