msp project evaluation lessons learned margaret phelps & kenneth w. hunter, sr. january 8, 2008...
TRANSCRIPT
MSP PROJECTEVALUATION
Lessons Learned
Margaret Phelps & Kenneth W. Hunter, Sr.
January 8, 2008
Millard Oakley STEM CenterMillard Oakley STEM Center
Our MSP History
Upper Cumberland Middle Grades Math Partnership (2004 - 2006)
TTU Engineering (7-12) Math Science Partnership (2005 - 2007)
Upper Cumberland Middle Grades Math Partnership (2007 - 2009)
Upper Cumberland Middle Grades Science Research Partnership (2008 - 2010)
Tennessee Pre-Engineering (7-12) Math Science Research Partnership (2008 - 2010)
Formative Assessment
Participant Surveys (Institutes & Workshops)
– Content– Format– Faculty– Scheduling– Facilities & food
Basis for change/
improvement
Formative Assessment
Daily faculty meetings during summer institutes
– Observations of master teachers• That was great! The teachers loved it!• Don’t ever do that again!
– Open discussion of issues• Extremely helpful for “non-education” faculty• Basis for “quick fix” improvements
Evaluation Designs
Experimental Design with Random Assignment– LEAs provide list of eligible teachers (or teams)– Local evaluator does random selection
Non-matched Comparison Groups– LEAs select teachers for intervention and control
groups– Groups are compared statistically on base line
measures to determine equivalence
Descriptive--No comparison group
Evaluation Components
Teachers– STEM Content Knowledge– STEM Pedagogy & Beliefs– Classroom Performance
Students– STEM Content Knowledge– STEM Authentic Assessment
Teacher Content--Praxis II
Advantages– External measure– Valid & Reliable– Tracks change over
time
Disadvantages– Cost– Obtaining score
reports– Inflexibility of
schedule
Instructor-made Test
Advantages– Matches institute
content– Aligns with standards– Control of testing
Disadvantages– Not necessarily valid
or reliable– Not normed– Forms might not be
equivalent
Purchased Test
Advantages– Control of testing– Access to scores
Disadvantages– Cost– Single form– Alignment
Teacher Pedagogy & Beliefs
Teacher-developed lesson modules
Reports on lesson implementation
Periodic self-assessment
Surveys– Beliefs and Attitudes (Math)– Misconceptions (Science)
Classroom Performance
Project-specific Observations – Conducted by LEA liaison or principal– Informs local leadership– Creates buy-in– Not valid for research purposes
Research-based Observations– Requires training of observers– Cost (perhaps)– Useful for research purposes
Student Content Knowledge
Middle school math & science– State testing program– Password-protected data available
(can be disaggregated for analysis)
– Compare NCE and proficiency level of students for pervious year (pre-) to year with project teacher (post-)
Student Content Knowledge
High school math & science– 8th grade scores are only common pre-test– HS scores not reported in NCE– Proficiency levels do not provide basis for
strong evaluation– Many courses not tested– ACT only available for graduates
Student Content Knowledge
Engineering Principles– JETS Assess
(Junior Engineer Technical Society)– Will be administered in classrooms– Fall/Spring for comparison– Control of data collection– Will also be administered to teachers
Student Authentic AssessmentElectronic Portfolio
Based on Engineering Principles
Multi-year – leads to pre-engineering certificate
Curricular components– Project Lead the Way– Math, Science & Career Technical courses &
projects
Student Authentic AssessmentElectronic Portfolio
Co-curricular components– FIRST LEGO League, FIRST Robotics– 4-H and Scouting projects– Work experience– Service learning– Utilize local opportunities
Contact Information
Dr. Margaret Phelps – [email protected]– All TTU-related MSP programs & Millard Oakley STEM Center
Mr. Kenneth W. Hunter, Sr. – [email protected]– Tennessee Pre-Engineering Math Science Research Partnership (2008-2010)
Dr. Susan Gore – [email protected]– Upper Cumberland Middle Grades Science Research Partnership (2008-2010)
Dr. Sandra Koczwara – [email protected]– Upper Cumberland Middle Grades Math Partnership (2004-2006)– Upper Cumberland Middle Grades Math Partnership (2007-2009)
Dr. Roy Loutzenheiser – [email protected]– TTU Engineering Math Science Partnership (2005-2007)
The instructional practices and assessments discussed or shown in these
presentations are not intended as an endorsement by the U.S. Department of
Education.