welcome to lib5080 the school library media program

Welcome to LIB5080The School Library Media Program

LIB 5080 The School Library Media Program

Today’s Learning• IMPACT Model – Guests in AETZ• Updates• Cross Program Collaboration• Program Planning and Evaluation

Cross Program Collaboration

Briefly describe the following:• Your CPC group and how you went about

the work• What did you learn …

– About 21st Century learning, schools, etc– About other programs/positions in K-16

education– About group processes

Planning & Evaluating Media Programs

Logic Models - Providing focus for planning and evaluating school library media programs

Based on SERVE Center’s Capacity for Applying Project Evaluation (CAPE) available online at http://www.serve.org/Evaluation/Capacity/

Why Plan and Evaluate?• Improve programs• Increase program effectiveness• Maintain funding• Generate additional funding• Enhance communication with stakeholders• Improve useful data collection• Inform decision makers• Increase staff and public awareness• Assess needs• Set program priorities• Focus and prioritize data needs

O’Sullivan, R. (2001). Utilization-focused evaluation. Presentation at the Siberian Evaluation Institute, Novosibirsk, Russia.

What is Evaluation?

• Evaluation is the systematic investigation of merit or worth.– Thoughtful, intentional, purposeful– Done for clear reasons– Organized and deliberate– Appraisal or judgment based on the best

evidence available

Guskey, T. (2005). Taking a second look at accountability: Strong evidence reflecting the benefits of professional development is more important than ever before. Journal of Staff Development. 26(1).

What is Formative Evaluation?• "When the cook tastes the soup,

that's formative evaluation. When the guests taste the soup that is summative evaluation." ~ Bob Stake

Cited in Patton, Michael Quinn (1997). Utilization-Focused Evaluation: The New Century Text. Edition 3. Thousand Oaks, CA: Sage Publishers, p. 69.

What is a Logic Model?

• A logic model is a graphic representation of the theory or logic behind a project or program.

• A logic model shows the forces that shape the program as well as the relationships among the key components.

Coffman, J. (1999). Learning from Logic Models. Cambridge, MA: Harvard Family

Research Project.

Logic Models

• Help to articulate the key elements

of a program

• Provide focus for program evaluation

• Can lead to program efficiency and effectiveness

• Promote stakeholder buy-in by helping clarify how the project works

Frameworks and Examples

• Harvard Family Research Council

• Kellogg Foundation

• National Science Foundation

• CAPE

Coffman, J. (1999). Learning from Logic Models. Cambridge, MA: Harvard Family Research Project. Retrieved

April 2, 2007 from http://www.gse.harvard.edu/%7Ehfrp/pubs/onlinepubs/rrb/learning.html

Harvard Family Research ProjectHarvard Family Research Project

Kellogg Foundation

W.K. Kellogg Foundation. (2001). Logic Model Development Guide. Battle Creek, MI: W.K. Kellogg Foundation. Retrieved April 2, 2007 from http://www.wkkf.org/Pubs/Tools/Evaluation/Pub3669.pdf

National Science Foundation

Frechtling Westat, J. (2002). The 2002 User-Friendly Handbook for Project Evaluation. Arlington, VA: National Science Foundation. Retrieved April 2, 2007 from http://www.nsf.gov/pubs/2002/nsf02057/nsf02057_3.pdf

Logic Map (Example) School Multi-year Technology Program

Inputs Objectives Goals

I. Students meet academic achievement standards

a. Policies & Mandates

b. Needs Assessment

c. District & State Plans

d. Research & Evaluation

4. Provide high quality professional development

for all staff members

A. Teachers develop and implement technology-

enhanced instruction and assessment.

B. Students use technology as an effective and

appropriate learning tool

e. Content Standards

f. Advisory Groups

II. Students meet standards for technology

proficiency

g. Planning Committee

h Experts

Strategies

5. Provide on-going, on-site curriculum and

instructional technology support to teachers

3. Provide sufficient technical and infrastructure

support staff

1. Provide leadership to support teaching and

learning with technology

This example of a logic map is provided for professional development and discussion purposes. It can be used as a starting point for developing a logic map for your particular technology program.

2. Provide ready access to sufficient technology and

media resources.

6. Develop and maintain school, family, community

partnership to support technology use in teaching

and learning.

Key Elements of a Logic Model

• Outcomes, Impacts, or Goals– Goals tell the ultimate purpose(s) of the

project or program. They answer the question “What difference will the project or program make in the long run?”

– In education, goals usually say what impact the program will have on learners and learning.


• Objectives, Intermediate Outcomes, or Short-term Goals

– Focus on program outcomes– Specify what the program will

accomplish– Answer the question “How will learning,

(teaching, the school, etc.) be improved as a result of this program?”

– Describe what you hope will happen as a result of the program

– Are measurable


• Strategies– Identify the major components of the

program– Are larger in scope than simple program

activities – Provide details about how the program

funds are spent– Describe what the program does– Are measurable


• Inputs or Contexts– Are the contexts or conditions that

influence program activities and strategies.

– Should be considered when planning program activities, e.g., policies, plans, needs assessment data.

STEM-ICT 3D Strategies Objectives/ Goals/

Intermediate Outcomes Long term

Outcomes

Middle and high school students are motivated to pursue ICT careers.

Teachers mentor others.

Design and deliver 3D curricular materials to diverse students.

Students develop virtual solutions to real world problems.

Students are prepared with the skills necessary to succeed in STEM.

Students teach teachers 3D technological skills.

Students mentor others.

Students collaborate via 3D virtual worlds.

Design and deliver teacher professional development on pedagogical strategies using virtual 3D worlds.

Teachers are engaged in continuous STEM learning.

Teachers build and sustain ongoing relationships with colleagues through interacting in the 3-D world about topics related to delivering content through the 3-D world.

Teachers engage students in the use of 3D world tools to explore solutions to real world problems.

Students pursue STEM careers.

Logic Model Exercise• Inside the packet are index cards labeled with components of a project. • Your tasks are as follows:

– organize these cards into the appropriate columns of the logic model template shown on the packet;

– tape the cards on the wall chart paper provided;

– connect the cards using marker or tape to make a logic model;

– review the connections between objectives and strategies using “if –then”.

Healthy Living Logic Model

Reflection

• Describe your process for creating the logic model.

• What worked? What didn’t?

Discussion

• What could be the advantages for creating and using a logic model for the school library media program?

• What might be the barriers or obstacles that would have to be overcome?

Potential Benefits

Logic models

• Provide clues to why the program does or does not work

• Can provide evidence for attributing outcomes

• Help improve programs– Expose faulty thinking– Develop a common language and

understanding of a program

Rogers, P., Petrosino, A., Huebner, T., & Hacsi, T. (2000). Program theory evaluation: Practice, promise, and problems. In Rogers, P., Hacsi, T., Petrosino, A., & Huebner, T. (Eds.) Program Theory in Evaluation: Challenges and Opportunities (pp. 5-13), San Francisco, CA: Jossey-Bass.

Used with permission ©ScienceCartoonPlus.com

Advantages to Using Logic Models• Teachers became more engaged and

reported greater program awareness (Finnan & Davis, 1995*)

• Staff understood the program better (Beckford, 1998*)

• Teachers were more actively involved in implementation and altered practices as a result of the evaluation (Darling, 1998*)

* As cited in Huebner, T. (2000). Theory-based evaluation: Gaining a shared understanding between school staff and evaluators. In Rogers, P., Hacsi, T., Petrosino, A., & Huebner, T. (Eds.) Program Theory in Evaluation: Challenges and Opportunities (pp. 79-89), San Francisco, CA: Jossey-Bass.

Logic Model

• Work in groups

• Develop a logic model for a school library media program

Sharing Logic Models

• Explain your logic

• What did you discover as you were creating your logic model?

What Happens Next?

• Define evaluation questions

• Determine benchmarks

• Select methods and measures

• Conduct the evaluation

• Analyze and draw inferences from data

• Modify the program

Writing Evaluation Questions

• Consider the purpose of your evaluation, what do you really care about?– Quality of implementation– Impact


• Evaluation questions about strategies ask how well the strategy is implemented.

• Evaluation questions about objectives ask about the impact of implementing the strategy.


• Try to avoid simple yes or no questions.

• Consider quantity questions (e.g., “how many,” “how much,” or “how often”).

• Consider quality questions (e.g., “how well,” “how effectively,” or “in what ways”).

• Be sure questions are self-contained – avoid putting implementation questions in terms of impacts.


• Examples– Implementation questions

• How many hours of sleep am I getting each week? (quantity)

• How soundly am I sleeping? (quality)

– Impact question

• How much weight have I lost? (quantity)

• How has my stress level changed? (quality)


• Using your logic model, select one strategy and brainstorm implementation questions that you might ask, to evaluate how well it is being accomplished.

• Select one objective and brainstorm impact questions that you might ask,

to evaluate how well it is being met.• Share and discuss.

Strategy:

Implementation Questions Data Sources/Analysis Methods

Benchmarks Decisions

What do you need to find out? How will you find out? What are your intermediate targets?

What decisions will you make based on the data?

Writing Evaluation Questions1. Insert a strategy from your logic model into the worksheet

1.

Strategy:






2. Write at least one implementation question for the strategy.

2.

Writing Evaluation Questions1. Insert an objective from your logic model into the worksheet

Objective:

Impact Questions Data Sources/Analysis Methods




1.


2. Write at least one impact question for the objective.

Objective:

Impact Questions Data Sources/Analysis Methods




2.

Data Sources/Analysis Methods• What do we mean by data?

– “Data” are pieces of information.– Common understanding is that data are

numbers – quantitative data or quantities of things.

– Data may also be qualitative – about qualities that are not easily measured in numbers.

– Both qualitative and quantitative data should be analyzed.

– Data are what we use to conduct formative evaluation of projects.

• What data do you need?– Your data needs are dictated by your

evaluation questions.– Ask yourself, “What information do I

need to answer this question?”– Gather all of the data you need, and only

the data you need, to answer your questions.

Data Sources/Analysis Methods

• Where do you get the data you need?– Check whether there is data already

available to you that might help answer your questions.

– Determine the data sources you might use to meet remaining data needs.

– Note that data sources are not data.• Example – teachers’ lessons plans are

a rich data source, but it is necessary to do something with them beforeyou have actual data.


• What are analysis methods?– The approaches and tools used to pull data

out of data sources– It may be helpful to think in four steps:

• Collecting data• Storing and organizing data• Analyzing data• Interpreting data

– Qualitative data may be quantified.


• What is “informal data?”– For an evaluation to be purposeful and

systematic, data must be relatively

formal, collected, stored and organized, and analyzed with some degree of rigor.

– Data not handled in this way may be thought of as “informal.”

– The trick is making informal data formal.


Strategy:





1. Review each evaluation question you wrote

1.


Strategy:





2. Identify the data source to answer the question.

2.


• Benchmarks– Are your targets.– Define levels of success.– Help you stop periodically to

examine progress.

Specifying Benchmarks


• Look at the Healthy Living logic model.

• Listen to the possible benchmarks.• What do you notice?

– How are data used to specify benchmarks?

– Are the expectations reasonable?– Do you think all the benchmarks will be

met?– Is it okay if the benchmarks aren’t met?


• Objective: Lose weight.• Benchmarking

– Baseline data• current weight (June 2007) = 165 lbs.

– Set target • weight 1 year from now (June 2008) = 145 lbs.

– Specify benchmarks • periodically stop and examine progress.


Specify benchmarks – periodically stop and examine progress.• By September, I will weigh 160 lbs.

• By December, I will weigh 155 lbs.

• By March, I will weigh 150 lbs.• By June, I will weigh 145 lbs.

• Benchmarks – one possible format (there are others…)


Possible Format: How many of who (or what) is going to do (or be) what

by when?

For Example: 70% of teachers will complete a collaborative unit with the media coordinator by mid-year.

Strategy:





1. Consider the questions and data sources you have written

1. 1.


Strategy:





2. Specify one or more benchmark for each datasource.

2.1.1.



• What baseline data do you have?

• What if you don’t have baseline data?

• In order to make adjustments you will want to collect, analyze, and compare data to benchmarks periodically• e.g. bi-monthly, monthly, each 9-weeks,

at the end of each semester


• Be realistic and reasonable.

• Consider how much time it will take to collect and analyze the data.

• What decisions might you make if evaluation findings show…

– Problems with implementation?– Good quality implementation?– Positive impacts (benchmarks

reached)?– Problems with impacts (benchmarks

not reached)?

• Who will make the decisions?

• Who will the decisions effect?

Making Decisions

Strategy:





1. Use the questions, data sources, and benchmarksyou have written ….

1.1.1.

Making Decisions

Strategy:





2. What decisions will you make based on your findings?

1.1.1.

Making Decisions

2.

Evaluation Plan Assignment• Work in your groups• Revise your logic model• Taking the role of a school MTAC,

select one objective the school will focus on for the school year.

• Complete an Objective worksheet for that objective.

• Complete a Strategy worksheet for all strategies associated with that objective.

Homework

• Work on evaluation plan

• Finish Special Topics Project

welcome to lib5080 the school library media program

Documents