educational evaluation as adaptive management€¦ · post-experience evaluation • evaluation...
TRANSCRIPT
![Page 1: Educational evaluation as adaptive management€¦ · post-experience evaluation • Evaluation only AFTER some experience fails to isolate the effect of the desired lesson • After](https://reader034.vdocument.in/reader034/viewer/2022050509/5f99e697db178568dd353170/html5/thumbnails/1.jpg)
Educational evaluation as adaptive management
Daniel T. [email protected]
www.eeb.ucla.edu/Faculty/Blumstein
Department of Ecology and Evolutionary BiologyUniversity of California Los Angeles
![Page 2: Educational evaluation as adaptive management€¦ · post-experience evaluation • Evaluation only AFTER some experience fails to isolate the effect of the desired lesson • After](https://reader034.vdocument.in/reader034/viewer/2022050509/5f99e697db178568dd353170/html5/thumbnails/2.jpg)
If we knew what it was we were doing, it would not be called research,
would it?
Albert Einstein
![Page 3: Educational evaluation as adaptive management€¦ · post-experience evaluation • Evaluation only AFTER some experience fails to isolate the effect of the desired lesson • After](https://reader034.vdocument.in/reader034/viewer/2022050509/5f99e697db178568dd353170/html5/thumbnails/3.jpg)
Are there bad decisions?
![Page 4: Educational evaluation as adaptive management€¦ · post-experience evaluation • Evaluation only AFTER some experience fails to isolate the effect of the desired lesson • After](https://reader034.vdocument.in/reader034/viewer/2022050509/5f99e697db178568dd353170/html5/thumbnails/4.jpg)
How could decisions be made?
www.errorik.com/archive/2004-11.htm
www.horaz.com/.../ARTICLES/Hussein_Saddam.htmhttp://www.state.gov/cms_images/20030320_bush-cabinet2_copy.jpg
•By edict from a dictator
•By a group of experts
•By a group of stakeholders and experts
•By popular vote
![Page 5: Educational evaluation as adaptive management€¦ · post-experience evaluation • Evaluation only AFTER some experience fails to isolate the effect of the desired lesson • After](https://reader034.vdocument.in/reader034/viewer/2022050509/5f99e697db178568dd353170/html5/thumbnails/5.jpg)
Decision making involves values
HSBC advertising campaign
![Page 6: Educational evaluation as adaptive management€¦ · post-experience evaluation • Evaluation only AFTER some experience fails to isolate the effect of the desired lesson • After](https://reader034.vdocument.in/reader034/viewer/2022050509/5f99e697db178568dd353170/html5/thumbnails/6.jpg)
What is the goal of decision making?
• To solve problems?
• To reach consensus?
• To go through a process?
• To make decision makers happy?
![Page 7: Educational evaluation as adaptive management€¦ · post-experience evaluation • Evaluation only AFTER some experience fails to isolate the effect of the desired lesson • After](https://reader034.vdocument.in/reader034/viewer/2022050509/5f99e697db178568dd353170/html5/thumbnails/7.jpg)
What is the goal of educational evaluation?
• To go through a process?
• To make supervisors happy?
• To comply with a law?
• To improve education?
www.enl.ips.k12.in.us/ID_Placement/j0399888.jpg
![Page 8: Educational evaluation as adaptive management€¦ · post-experience evaluation • Evaluation only AFTER some experience fails to isolate the effect of the desired lesson • After](https://reader034.vdocument.in/reader034/viewer/2022050509/5f99e697db178568dd353170/html5/thumbnails/8.jpg)
Adaptive management
• Active adaptive management: management plans are modified based on the results of well-designed experiments that collect data on factors or variables that are demonstrably important for conservation or management
• Passive adaptive management: managers use historical data or data from uncontrolled experiments to come up with “best guess” management recommendations, the fate of which may be studied.
![Page 9: Educational evaluation as adaptive management€¦ · post-experience evaluation • Evaluation only AFTER some experience fails to isolate the effect of the desired lesson • After](https://reader034.vdocument.in/reader034/viewer/2022050509/5f99e697db178568dd353170/html5/thumbnails/9.jpg)
Controls in active adaptive management
• Essential
• After-Before, Treatment-Control designs are particularly effective
• They allow effect of manipulation to be isolated
• They control for ontogenetic change
• Design comes from ‘learning’ experiments
![Page 10: Educational evaluation as adaptive management€¦ · post-experience evaluation • Evaluation only AFTER some experience fails to isolate the effect of the desired lesson • After](https://reader034.vdocument.in/reader034/viewer/2022050509/5f99e697db178568dd353170/html5/thumbnails/10.jpg)
Controls in active adaptive management
• Not routinely used– Too few animals– Many factors may have to be manipulated– Ethical issues: mortality risk associated with
control groups– Lack of familiarity with experimental design
• Rather, managers use conventional wisdom, or monitor and modify their management activities accordingly
![Page 11: Educational evaluation as adaptive management€¦ · post-experience evaluation • Evaluation only AFTER some experience fails to isolate the effect of the desired lesson • After](https://reader034.vdocument.in/reader034/viewer/2022050509/5f99e697db178568dd353170/html5/thumbnails/11.jpg)
Educational evaluation as adaptive management
• Goal: To effectively optimize teaching
• Process: Conduct a ‘learning experiment’
http://www.groundwork-se.org.uk/upload/images/image32.jpg
![Page 12: Educational evaluation as adaptive management€¦ · post-experience evaluation • Evaluation only AFTER some experience fails to isolate the effect of the desired lesson • After](https://reader034.vdocument.in/reader034/viewer/2022050509/5f99e697db178568dd353170/html5/thumbnails/12.jpg)
How to conduct a learning experiment
• Classroom T (for treatment) is the one exposed to the educational experience
• Classroom C (for control) is not• A better ‘learning’ control is to have them have
the same experiences, but not paired, this is often difficult to achieve in an educational situation
• Key comparison is the difference after-before compared across the treatment and control
![Page 13: Educational evaluation as adaptive management€¦ · post-experience evaluation • Evaluation only AFTER some experience fails to isolate the effect of the desired lesson • After](https://reader034.vdocument.in/reader034/viewer/2022050509/5f99e697db178568dd353170/html5/thumbnails/13.jpg)
After-Before compared across Treatment & Control
Control: 4-3 = 1Treatment: 10-3 = 7
7:1 is a large effectof the educationalintervention
![Page 14: Educational evaluation as adaptive management€¦ · post-experience evaluation • Evaluation only AFTER some experience fails to isolate the effect of the desired lesson • After](https://reader034.vdocument.in/reader034/viewer/2022050509/5f99e697db178568dd353170/html5/thumbnails/14.jpg)
After – Before, Treatment-Control designs are superior to
post-experience evaluation
• Evaluation only AFTER some experience fails to isolate the effect of the desired lesson
• After – Before only misses the idea that students are always learning– i.e., you have to control for other
experiences and ontogenetic change
• Without a control, it’s impossible to isolate the effect of the lesson
![Page 15: Educational evaluation as adaptive management€¦ · post-experience evaluation • Evaluation only AFTER some experience fails to isolate the effect of the desired lesson • After](https://reader034.vdocument.in/reader034/viewer/2022050509/5f99e697db178568dd353170/html5/thumbnails/15.jpg)
The ‘spacing effect’: a lesson in impediments to application
• Spacing effect: for a given amount of study time, studying in small pulses is much better than cramming.
• One of the most robust findings from decades of research in learning and experimental psychology
• Not widely used
![Page 16: Educational evaluation as adaptive management€¦ · post-experience evaluation • Evaluation only AFTER some experience fails to isolate the effect of the desired lesson • After](https://reader034.vdocument.in/reader034/viewer/2022050509/5f99e697db178568dd353170/html5/thumbnails/16.jpg)
Some impediments to application (modified from Dempster 1988 Am. Psych.)
• The phenomenon has not been known• The phenomenon has not received recent
documentation• The phenomenon cannot be linked to issues of
current concern to educators• The phenomenon has not been demonstrated
satisfactorily in school-like activities• The phenomenon has not been demonstrated
satisfactorily in the classroom• The phenomenon is not sufficiently understood
![Page 17: Educational evaluation as adaptive management€¦ · post-experience evaluation • Evaluation only AFTER some experience fails to isolate the effect of the desired lesson • After](https://reader034.vdocument.in/reader034/viewer/2022050509/5f99e697db178568dd353170/html5/thumbnails/17.jpg)
Why are ‘learning experiments’ not often performed?
• Don’t know about them
• Never done them before
• No interest/will/ability to have proper control
• Lack of clearly identified learning objectives
![Page 18: Educational evaluation as adaptive management€¦ · post-experience evaluation • Evaluation only AFTER some experience fails to isolate the effect of the desired lesson • After](https://reader034.vdocument.in/reader034/viewer/2022050509/5f99e697db178568dd353170/html5/thumbnails/18.jpg)
Learning objectives• What are the key outcomes of an educational
experience?– Simply to provide an experience
•no evaluation required– To teach
•What?
• Once goals/objectives/outcomes are identified, then it’s possible to design proper evaluation
• An after-before, treatment vs. control design will help efficiently and effectively improve environmental education
![Page 19: Educational evaluation as adaptive management€¦ · post-experience evaluation • Evaluation only AFTER some experience fails to isolate the effect of the desired lesson • After](https://reader034.vdocument.in/reader034/viewer/2022050509/5f99e697db178568dd353170/html5/thumbnails/19.jpg)
Life is like a sewer, what you get out of it depends on what you
put into it.
Tom Lehrer
![Page 20: Educational evaluation as adaptive management€¦ · post-experience evaluation • Evaluation only AFTER some experience fails to isolate the effect of the desired lesson • After](https://reader034.vdocument.in/reader034/viewer/2022050509/5f99e697db178568dd353170/html5/thumbnails/20.jpg)
Asking the right questions
• It’s hard
• It’s value laden
• It’s political
• It’s essential
http://sivinkit.net/archives/question1.jpg
![Page 21: Educational evaluation as adaptive management€¦ · post-experience evaluation • Evaluation only AFTER some experience fails to isolate the effect of the desired lesson • After](https://reader034.vdocument.in/reader034/viewer/2022050509/5f99e697db178568dd353170/html5/thumbnails/21.jpg)
What are the right questions?
• Those that evaluate the learning objectives
• Ideally, learning objectives will enable students to make sustainable decisions when faced with complex, real-world data
• Environmental education should be much more than simply teaching environmental awareness, it should teach citizenship, awareness, and sustainability (Blumstein & Saylan 2007 PLoS-Biology)
![Page 22: Educational evaluation as adaptive management€¦ · post-experience evaluation • Evaluation only AFTER some experience fails to isolate the effect of the desired lesson • After](https://reader034.vdocument.in/reader034/viewer/2022050509/5f99e697db178568dd353170/html5/thumbnails/22.jpg)
Conclusions• Proper evaluation is an effective way to improve
education and meet learning objectives
• After-Before, Treatment vs. Control designs are a powerful evaluation tool
• Many disciplines are reluctant to use such evidence-based approaches for evaluation
• The crux of the matter is developing the right learning objectives and then asking the right questions to evaluate them