Evaluation Test

Justin K. ReeveEDTECH 505-4173Dr. Ross Perkins

Reflection

It's useful to know the different models that are available for evaluation, and being able to identify which one is best for a situation. The goals, activities, and evaluation procedures in the programs play a crucial role in how the evaluation is conducted. Efficiency, efficacy, and impact should be measured across any program. There are a number of programs in our district that currently lack any evaluation, or for which we collect a minimal amount of information that doesn't follow a defined, systematic process. At best, a simple survey is supplied at the end of the programs (for example, see http://wsd.net/evolve-evaluation), but no formative evaluation is conducted during the training sessions themselves, and the survey results are never reviewed by the training coordinator. Until these deficiencies are resolved, our evaluation process is ineffective. The knowledge I've learned in the class so far has given me the ability to identify these weak points.

Rationale for Evaluation

An evaluation produces statistical data that helps sponsors and staff members choose appropriate resources, identify areas in need of adjustment, become aware of program strengths and weaknesses, and understand the outcome of actions conducted in a program.

Photo Source: http://bit.ly/aFQPDC

Evaluation and EdTech

Evaluation enables educational technologists to choose, improve, and measure the instruction and learning process. Adopting a defined, systematic process gives the evaluator a vision of the aspects of a program working together.

Photo Source: http://www.flickr.com/photos/17258892@N05/2588347668/

Research and Evaluation

Research uses systematic processes to determine general knowledge, whether through controlled variables and the scientific method, or qualitative methods that have a rigorous base in scientific inquiry. Evaluation allows the evaluator and program’s stakeholders to better understand the internal processes of the program.

Exterior of Watch (Research) Photo Source: http://bit.ly/dkUdot Interior of Watch (Evaluation) Photo Source: http://bit.ly/cTrz6z

Efficiency, Efficacy, and Impact

Efficiency refers to the balance between time and resources spent on a program, and how well the returns exceed the investment.

Efficacy refers to how well objectives and desired needs are attained.

Impact refers to how participants change their behavior on a permanent basis.

Scales Photo Source: http://bit.ly/bd3bmwEnergy-Efficient Light Bulb Photo Source: http://bit.ly/a6tYvNChange Photo Source: http://bit.ly/9tqUUv

Impact of Evaluation

An evaluation can identify connections between goals and needs and implement focused activities to these ends. Formative and summative evaluation results identify processes that work effectively, and decisions that can be made for necessary program modifications.

Photo Source: Created by Justin K. Reeve

Goal-Free Model

A data collection- and analysis-oriented model, ideal for qualitative evaluations, in which program aspects come from observation of the program, without regard to goals and objectives. The evaluator uses the data to infer conclusions of the program's impact on clients' needs.

Photo Source: http://arcmit01.uncw.edu/jonesi/Images/ScrivenGoalFreeEvaluation.png

Kirkpatrick's Four-Level Model

A model in which students evaluate the program's objectives, value, and relevance (Reaction), and participate in objective-based assessments (Learning). The model evaluates the impact of the program on participants (Behavior) and the organization for the decision-makers (Results).

Photo Source: Created by Justin K. Reeve

Qualitative vs. Quantitative

Qualitative data are collected with observations, interviews, surveys, and case studies. Data are subjectively interpreted. Quantitative data are collected with tests, instruments, and other quantifiable measuring tools, and clearly state numerical facts and may be used to predict outcomes.

Photo Source: Created by Justin K. Reeve

Levels of Data

Nominal data consists of mutually exclusive unordered categories (black/white).

Ordinal data consists of ranked, ordered items, without numerical differences between the items (top 10 list).Interval data is ranked numerically without an absolute zero on the scale (temperature).  Ratio data is ranked numerically with an absolute zero on the scale (length).

Ordinal Photo Source: http://bit.ly/9QSadTRatio Photo Source: http://bit.ly/aDQ5pM

Nominal Photo Source: Created by J. ReeveInterval Photo Source: http://bit.ly/90pIEc

Data Instruments

Interview: A conversation which obtain considerable information, but with poor comparison across instances.Scale: A response in ranked intervals; a scale should be well-organized to prevent evaluator bias.Test: An assessment tool which produces numerical data. Observation: Constructing narratives of perception from predetermined criteria.

Interview Photo Source: http://bit.ly/ddPXQxScale Photo Source: Personal Collection

Test Photo Source: http://bit.ly/bfrQbWObservation Photo Source: http://bit.ly/aXQawR

Formative vs. Summative

Formative evaluation tracks a program as it happens. Real-time adjustments can be made to the program from these results. Summative evaluation occurs after the program and gauges the overall effectiveness and objective attainment.

Parent Assisting with Homework Photo Source: http://is.gd/frHYnParent Reviewing Homework Photo Source: http://is.gd/frHRF

Samples and Populations

The population represents all potential participants that meet the criteria for the intended evaluation. The sample is a selection of participants, typically random, that is representative of the target population, or represents a desired section of the population.

Photo Source: http://is.gd/frvgb

Validity and Reliability

Reliability refers to data consistency, and the ability to produce the same results when an analysis is repeated. Validity refers to data where an accurate conclusion follows from the premises. Evaluation data should be both reliable and valid.

Photo Source: Created by Justin K. Reeve

Independent and Dependent

Variables are elements of an evaluation that may take on different characteristics or values. Dependent variables are uncontrolled, and strictly observed or studied. Independent variables are controlled by the evaluator or participants, and manipulated when necessary.

Photo Source: http://www.flickr.com/photos/tentangqatar/4385556714/

Criterion- vs. Norm-Referenced

Criterion-referenced tests measure individual learning and personal achievement against curricular standards, with clear expectations of participant performance.

Norm-referenced tests rank how students compare to others in the population, without setting goals or using standards of achievement.

Photo Sources: http://bit.ly/b2Tv07     http://bit.ly/c2CQgf

Activity Evaluation

The idea of using visual metaphors as an assessment is an interesting one. I'd actually like to see some data comparing the benefits and disadvantages of this form of assessment vs. a traditional test. I personally struggle a little with coming up with actual metaphors and not just visual examples of the concepts being assessed (hopefully this isn't too evident in these slides), but that gives me a chance to improve my abstract thinking skills. I would like it if the activity's instructions explicitly gave extra allowed word space to explain each visual metaphor on the following slide, so I could better justify my choice of images (though maybe this defeats the point of coming up with a strong visual metaphor).