an overview of evidence of effectiveness vs evidence of need€¦ · •an overview of evidence of...

• An overview of evidence of effectiveness vs evidence of need

• An overview of the range of evidence that can be used to

demonstrate the effectiveness of your initiative’s approach

• Examples of how organizations have successfully used evidence to

support their applications

• Tips on where to search for external arts, culture and heritage

evidence

Strong Application

Activitiestied to

evidence

Good evidence

(quality)

Relevant evidence

(i.e. Grant Result)The best applications use good quality evidence that is relevant to their project, and aligns with

the chosen Grant Result

Example:

Applicants must show evidence that the

model/approach you wish to use is proven

to be effective in achieving its objective

1. Program efficacy

2. Evidence-based approach

Applicants should articulate why the

program is needed in the community they

are working in

1. Evidence of need

2. Evidence of demand for program or approach in

the community

Don’t confuse evidence of need with evidence of effectiveness:

Solid evidence of

effectiveness

Solid evidence of need

Both have their own value, but are distinct in an application!

Example: A community-based performing arts program for racialized youth aims to help them foster a career in the arts

Data demonstrating program effectiveness

• Survey data demonstrating specific positive change

• Interview or focus group data showing positive effects

• Observed changes by staff, using the Observation Method

Data that does not demonstrate effectiveness

• Anecdotes from participants

• Comments about satisfaction

• # people that participated

• Non-specific or vague survey data

Data demonstrates need

• Lack of access to programming locally

• Low rates of associated skills in youth

• Community consultations on need for programming

Building on outside evidence

Documenting evidence of your program’s

effectiveness

Leveraging relevant evidence from academic or community research or evaluations from other

organizations/programs

Using your own evidence that has been generated through evaluation

8

• Evaluation is the systematic gathering and interpretation of information about a program, activity, policy or needo Results from evaluations can help us learn about what's working, and what isn’t, and can

help us make decisions based on our own evidence. Evaluation is one way of generating evidence!

• Research is the systematic investigation and study of materials and sources in order to establish facts and reach new conclusionso Research is also a way of exploring or investigating something. It’s about establishing

facts and drawing conclusions from those facts. Research is one way of generating evidence!

There are many kinds of evidence, but the way we collect it and the purpose behind it are distinct.

Systematic reviews or meta-analyses• Reviews an entire

body of research on a particular program or method

• Summarizesevidence both for and against

• Provides an estimate of the size of impact

Program Evaluations

Vary greatly in terms of method, approach, rigour, context

Grey Literature

Share similar methodologies with research studies, mostly not academically reviewed

Example – Literature Review produced by United Way on youth engagement

Individual research studies• Experiments and

quasi-experiments• Longitudinal studies

with pre- and post-measures

• Typically peer-reviewed, scrutinized by other topic experts; may be less biased

• May have issues of validity and objectivity

Traditional Knowledge

• Orally transmitted knowledge, innovations and practices of Indigenous communities

• From experience gained over the centuries and adapted to the local culture and environment

INSPIRED PEOPLE

Better quality programming and infrastructure to

experience culture, heritage and the arts

Skills and knowledge are transferred to the next generation of artistic leaders

More people connect with culture, heritage and the arts

Access to arts-based learning opportunities and compelling artistic,

cultural and heritage experiences

People are engaged in community-based arts

creation

Preservation and animation of

cultural heritage

12

Grant Result: Skills and knowledge are transferred to the next generation of artistic leaders

The Project:

• Building on the success of a proven model, they are establishing low-barrier digital creator spaces for youth in six communities across Northern Ontario

• 3 year grant, $700,000• Population: General• Age Group: Youth (adolescents 15 to18)• Community Size: Rural or Small Communities (20,000 or less)

13

What type of evidence is it?

• Grey Literature• A study using developmental evaluation produced the University of Chicago

What was the research question?

What was the motivation of the research?

What were the findings?

Used external evidence to demonstrate a proven model “Teens, Digital Media, and the Chicago Public Library”, The University of Chicago Consortium on Chicago School Research, 2013.

• Answer key questions about the YouMedia program at Chicago Public Library• Help inform the design of similar initiatives to engage youth in learning

through technology and youth-centered activities

• How does the design of YOUmedia shape youth participation?• What benefits do youth perceive from their participation? • How do youth engage with media? What role to adults play in engaging teens?

• The model provides teens with a sense of belonging, supports teens’ interests, self-reported improvement in digital media skills

• Successful replication will depend on both local and universal conditions, such as strong partnerships, the right space and adequate resources for equipment

14

• Relevant to their project (promising model)

• Aligned with the Grant Result (skills & knowledge transfer to the next generation of artistic leaders)

• Leverages external evidence (via developmental evaluation)

What makes this a good demonstration of evidence in an application?

15

Grant Result: Access to arts-based learning opportunities and compelling artistic, cultural and heritage experiences

The Project:

• A creative writing workshop expansion project to provide under-served children with the inspiration and support to become confident and creative storymakers through writing, visual arts and digital media

• 3 year grant, $150,000• Population: Diverse cultural communities• Age Group: Children up to 12 years• Community Size: Urban Centres and Metropolitan Suburbs (100,000+)

16

Developed its own preliminary results through evaluation

Collected quantitative & qualitative data from students & teachers via a created survey

• Increased skill & knowledge development (87% students)• Increased interest in writing (92% teachers)• Self-confidence in writing and art• Enthusiasm for future story making (93% students)

How was evidence collected?

What did they wish to discover?

Wanted to understand extent to which students felt inspired and supported to become confident and creative story-makers through writing, visual arts and digital media

What did their preliminary results state?

17

Used external evidence to demonstrate promising practice Something to Say: Success Principles for Afterschool Arts Programs From Urban Youth and Other Experts, Wallace Foundation, 2013.


• Grey Literature• Research commissioned by the Wallace Foundation• Literature review


“How do nonprofit organizations serving urban, low-income communities develop or refine effective, engaging and accessible arts programs?”


To provide a model of practices that can provide a blueprint for community-based organizations to emulate, so proven approaches can be deployed and more often


“10 Principles for Effective, High-Quality Out of School Time Arts Programs”Example: Principle #4: There is a culture of high expectations, respect for creative

expression and an affirmation of youth participants as artists.

18

• Relevant to the project (promising model)

• Aligned with the Grant Result (access to arts-based learning)

• Promising preliminary results

• Leveraged both internal and external evidence


19

Grant Result: Preservation and animation of cultural heritage

The Project:

• Growing and enhancing cultural programming at an Indigenous community cultural centre (cultural counselling, facilitating sweats and other ceremonies, collecting oral histories, language classes)

• 3 year grant, $750,000• Population: Indigenous• Age Group: General• Community Size: Urban Centres and Metropolitan Suburbs (100,000+)

20

Used external evidence to demonstrate promising practice (1) “Creative Arts, Culture, and Healing: Building An Evidence Base”, Pimatisiwin: A Journal of Aboriginal and Indigenous Community Health 8(3) 2010


• Grey Literature• Research commissioned by Aboriginal Health Foundation of 137 FNMI

projects they funded• surveys & key informant interviews


“What happens when art, music, dance, storytelling and other creative arts become a part of healing programs?”


Addressing a gap in program evaluation data on the role the arts play in healing programs


Evidence that creative arts, culture and healing are linked:• Eg. Reconnecting to cultural traditions through practice of traditional creative

arts can contribute significantly to healing (drumming for male youth)

21

Used external evidence to demonstrate promising practice (2) “Closing the Gap: Supporting healthy communities through arts programs”, Resource sheet no.28 produced for the Closing the Gap Clearinghouse, Australian Institute of Health and Welfare 2014


• Grey Literature• Research paper produced by Australian government • Literature Review


• Why are arts activities important in Indigenous communities?• What are the benefits of participating in arts programs?• How do arts programs support healthy Indigenous communities?


Review the available evidence of a range of arts programs in relation to their effects on supporting and building healthy communities


Evidence of beneficial impacts and outcomes from arts programs in Indigenous communities + principles for implementing effective arts programs in Indigenous communities

22

• Relevant to their initiative (effectiveness of approach)

• Aligned with the Grant Result (preservation & animation of culture)

• Leveraged two external pieces of grey literature


23

Testimonials/Letters of Support

Program Satisfaction Surveys

Attendance lists

Reports that describe the need

Posters and festival guides

Emails regarding revenue levels

Lesson Plans/Workshop agendas

Strategic Plans

Quotes

Examples of evidence that do not demonstrate effectiveness:

Relevance

• How similar is the context of the study to your community? Can findings be generalized?

• Are the study or evaluation’s goals aligned with your project’s intended outcomes?

• If looking at an older study, is it still relevant in today’s context?

Quality

• Were good research/evaluation methods used? Consider how data was collected and analyzed, what the period of time was, and if a comparison group was used.

• How big was the research sample, and was it representative?

• What was the extent of desired change?

• Consider if other changes/policies/programs are responsible for changes?

• Who produced the study/evaluation, and why?

27

Additional Search Tips:

• Don’t stick to one study; try to find multiple pieces of research• Look at guiding documents from your sector • Search for pieces of research that have done a systematic review of available

research (e.g. literature reviews, meta analyses)

Additional Application Tips:

Don’t make it a challenge for the people assessing your application!

• Properly cite the research you are using • Make sure to attach it to your application, highlight important areas/key

findings that relate to your project and the Grant Result

o Be critical!

o Evidence should inform your expertise, not replace it

o Think about the value evidence can bring to informing your work generally, not just in a grant application

o The onus is on you to demonstrate that your project is informed by relevant and good quality evidence

o If don’t have internal evidence, consider applying up to 10% of your application budget toward doing your own evaluation.

30

OTF’s online Evaluation E-Learning coursehttps://otf.ca/knowledge/resources/evaluation-e-learning-course

Arts Evaluation Webinars on the Knowledge Centre:

What makes ARTS FOR CHANGE work EXCELLENT with Animating Democracy

How do ARTS & CULTURE make a DIFFERENCE with Animating Democracy

Evaluating Arts for Social Change & Community-Engaged Arts Projects with the International Centre for Arts for Social Change

http://bit.ly/OTFGrow

31

http://bit.ly/OTFGrow

an overview of evidence of effectiveness vs evidence of need€¦ · •an overview of evidence of...

Documents