Slide 1

Social Justice Amidst Standards and Accountability Donna M. Mertens, Keynote Independent Consultant Minnesota Evaluation Studies Institute, MESI Spring Training Minneapolis MN March 2015

Slide 2

Why this topic? Why now? Social Justice Amidst Standards and Accountability 3/11/2015 Mertens Keynote MN MESI March 2015 2

Slide 3

2015 International Year of Evaluation EvalPartners, UNICEF, UN Women, and the International Organization for Cooperation in Evaluation Peer 2 Peer Initiative gender and equity focused evaluation 3/11/2015 Mertens Keynote MN MESI March 2015 3

Slide 4

Need for Social Justice Link to Evaluation: Pervasiveness of racism and other isms 3/11/2015 Mertens Keynote MN MESI March 2015 4

Slide 5

Need for a Social Justice Lens Marie Battiste (2000): society is sorely in need of what Aboriginal knowledge has to offer Chilisa (2005) it is an issue of life and death Brookes (2006) Failure to include racism as a potential contributor to disparities 3/11/2015 Mertens Keynote MN MESI March 2015 5

Slide 6

My hypothesis: If we begin by prioritizing social justice and human rights And we appropriately involve community members in the evaluation process, Then we will increase the probability of social transformation as a result of our evaluation. 3/11/2015 Mertens Keynote MN MESI March 2015 6

Slide 7

What does our evidence lack in credibility from a social justice perspective? 3/11/2015 Mertens Keynote MN MESI March 2015 7

Slide 8

Social Justice Standards Accountability 3/11/2015 Mertens Keynote MN MESI March 2015 8

Slide 9

Accountability Question Did it work? Should we keep spending our money on it? 3/11/2015 Mertens Keynote MN MESI March 2015 9

Slide 10

Social Justice Questions Was it the right thing? Was it chosen and/or developed and implemented in culturally responsive ways? Were contextual issues of culture, race/ethnicity, gender, disability, deafness, religion, language, immigrant or refugee status, age or other dimensions of diversity used as a basis for discrimination and oppression addressed? How were issues of power addressed? Do we want to continue to spend money on things that dont work? 3/11/2015 Mertens Keynote MN MESI March 2015 10

Slide 11

Resources for Evaluators AEA Guiding Principles AEA Public Statement on Cultural Competency Joint Committee Program Evaluation Standards Evaluators who have dedicated their lives towards furthering social justice Communities who want social justice 3/11/2015 Mertens Keynote MN MESI March 2015 11

Slide 12

Good news; bad news for social justice in evaluation 1.2% (n=22) in Harners (2014) survey of evaluators identified their theoretical frame as Social Justice 69% (n=819) of 1,187 evaluators either strongly or somewhat agreed with this statement: Evaluation should focus on bringing about social justice. 3/11/2015 Mertens Keynote MN MESI March 2015 12

Slide 13

AEA Guiding Principles (2004): Systematic Inquiry, Competence, Integrity/Honesty, Respect for People, and Responsibilities for General and Public Welfare To ensure recognition, accurate interpretation and respect for diversity, evaluators should ensure that the members of the evaluation team collectively demonstrate cultural competence. Cultural competence would be reflected in evaluators seeking awareness of their own culturally-based assumptions, their understanding of the worldviews of culturally-different participants and stakeholders in the evaluation, and the use of appropriate evaluation strategies and skills in working with culturally different groups. Diversity may be in terms of race, ethnicity, gender, religion, socio-economics, or other factors pertinent to the evaluation context. (Competence B2) 3/11/2015 Mertens Keynote MN MESI March 2015 13

Slide 14

AEA Guiding Principle: Cultural Competence Linkage with Transformative Axiology and Epistemology To ensure recognition, accurate interpretation, and respect for diversity, evaluators should ensure that the members of the evaluation team collectively demonstrate cultural competence (American Evaluation Association, Public Statement on Cultural Competence in Evaluation, 2011) 3/11/2015 Mertens Keynote MN MESI March 2015 14

Slide 15

What is cultural competence? Cultural competence is not a state at which one arrives; rather, it is a process of learning, unlearning, and relearning. It is a sensibility cultivated throughout a lifetime. Cultural competence requires awareness of self, reflection on ones own cultural position, awareness of others positions, and the ability to interact genuinely and respectfully with others. Culturally competent evaluators refrain from assuming they fully understand the perspectives of stakeholders whose backgrounds differ from their own. Cultural competence is context dependent. (AEA 2011) 3/11/2015 Mertens Keynote MN MESI March 2015 15

Slide 16

Evaluation and Validity and Cultural Competence accurately and respectfully reflect the life experiences and perspectives of program participants in their evaluations. establish relationships that support trustworthy communication among all participants in the evaluation process. draw upon culturally relevant, and in some cases culturally specific, theory in the design of the evaluation and the interpretation of findings. select and implement design options and measurement strategies in ways that are compatible with the cultural context of the study. (AEA 2004, p. 6) 3/11/2015 Mertens Keynote MN MESI March 2015 16

Slide 17

Recognize the dynamics of power Cultural groupings are ascribed differential status and power, with some holding privilege that they may not be aware of and some being relegated to the status of other. For example, language dialect and accent can be used to determine the status, privilege, and access to resources of groups. (AEA, 2004, p. 7) 3/11/2015 Mertens Keynote MN MESI March 2015 17

Slide 18

Standards for Good Evaluation (Yarborough et al. 2011) Utility: evaluator credibility, attention to stakeholders, negotiated purposes, explicit purposes Feasibility: management, practical, contextual viability (recognize, monitor, and balance the cultural and political interests and needs of individuals and groups) Propriety: responsive and inclusive, formal agreements, human rights and respect, clarity and fairness, transparency and disclosure. Accuracy: valid, reliable, explicit program and context descriptions, sound designs and analysis Evaluation accountability: meta-evaluation 3/11/2015 Mertens Keynote MN MESI March 2015 18

Slide 19

Accountability and Social Justice Code word for RCT? Accountable to whom? Ways to strengthen arguments about accountability are not limited to RCTs Theoretical streams in evaluation provide options for demonstrating accountability 3/11/2015 Mertens Keynote MN MESI March 2015 19

Slide 20

Evaluation Theory and Social Justice Alkins Tree: Methods, Use & Values Branches Mertens Keynote MN MESI March 201520 ValuesMethods Use 3/11/2015

Slide 21

Mertens & Wilson 2012: Methods, Use, Values, & Social Justice Branches Mertens Keynote MN MESI March 201521 Methods Use Values Social Justice 3/11/2015

Slide 22

Metaphor: Tree or Water? Mertens Keynote MN MESI March 2015 22 3/11/2015

Slide 23

Paradigms & Branches ParadigmBranch Post-positivistMethods ConstructivistValues TransformativeSocial Justice PragmaticUse Mertens Keynote MN MESI March 2015 23 3/11/2015

Slide 24

Transformative Theories Feminist theories Critical Race Theories Critical Theories Human Rights Theories Disability Rights Theories Transformative Participatory Action theories Indigenous theories Deafness rights theories Queer theories 3/11/2015 Mertens Keynote MN MESI March 2015 24

Slide 25

Transformative Paradigm PhilosophicalAssumptions AxiologyRespect for cultural norms; support for human rights and social justice; reciprocity OntologyIssues of power & critical interrogation of multiple realities: social, political, cultural, economic, race/ethnic, gender, age, religion and disability values to unmask those that sustain an oppressive status quo EpistemologyIssues of power & Interactive link; knowledge is socially and historically located; trusting relationship. Methodology Qualitative (dialogic)/ Quantitative / Mixed Methods; Context Mertens Keynote MN MESI March 2015 25 3/11/2015

Slide 26

Court Access Project Start from community Advisory board formed Begin understandings from the vantage point of the least privileged Work with an awareness of diversity in the community Provide respectful support for engagement Use a cyclical approach to data collection and use 3/11/2015 Mertens Keynote MN MESI March 2015 26

Slide 27

Example: HIV/AIDS Prevention in Botswana Botswana youth: addressing power inequities in the fight against HIV/AIDS using a transformative lens Mertens Keynote MN MESI March 2015 27 3/11/2015

Slide 28

Transformative Cyclical Mixed Methods Design Mertens Keynote MN MESI March 2015 28 Stage 4 Sequential Stage 5 Concurrent Stage 3 Sequential Stage 2 Concurrent Stage 1 Qual Assemble team; read documents; engage in dialogues Preliminary studies: youth, older men Process eval Pilot intervention: Observations, Interviews, Surveys Demographic information; Surveys; Incidence data Pretest: Knowledge, Attitude, Behavior; Post tests: Quant Qual; Behavior & Policy Change; Transfer To other contexts RCT 3/11/2015

Slide 29

Indigenous Contributions Social justice as a frame for evaluation Engagement with indigenous communities 3/11/2015 Mertens Keynote MN MESI March 2015 29

Slide 30

Indigenous Paradigm Chilisa (2012) Wilson (2008) Linda T. Smith (1999) Marie Battiste (2000) LaFrance & Crazy Horse (2009) Relational Axiology Relational Ontology Relational Epistemology Relational Methodology (Chilisa, 2012) 3/11/2015 Mertens Keynote MN MESI March 2015 30

Slide 31

Ethics and Indigeneity Connections between people, past, present, and future, all living and nonliving things (ubuntu) Respect and reciprocity: listens, pays attention, acknowledged, and crates space for the voices and knowledge systems of Indigenous people Contribute to a better future Spirituality 3/11/2015 Mertens Keynote MN MESI March 2015 31

Slide 32

Questions What is credible evidence? What criteria establish credibility of evidence? What does our evidence lack in credibility? What sources give us insight into credibility? What is the place of social justice in credibility? How do voices of marginalized communities enhance our understandings of credibility? Why is it important for researchers to understand marginalized communities perspectives of social justice? Mertens Keynote MN MESI March 2015 3/11/2015 32

Slide 33

Methodological Questions What data collection methods can the evaluator use to become better acquainted with the various stakeholder groups? How can the evaluator design the study so that the community members are included in decision making? How could the use of mixed methods contribute to the quality of the study? How can the evaluator design the methods to increase the probability of furthering social justice? 3/11/2015 Mertens Keynote MN MESI March 2015 33

Slide 34

Water as Metaphor Eventually, all things merge into one, and a river runs through it. The river was cut by the world's great flood and runs over rocks from the basement of time. - Norman Fitzroy Maclean When you put your hand in a flowing stream, you touch the last that has gone before and the first of what is still to come. - Leonardo da Vinci My soul is full of longing For the secret of the Sea, And the heart of the great ocean Sends a thrilling pulse through me Henry Wadsworth Longfellow http://www.finestquotes.com/select_quote-category-Water-page-1.htm#ixzz2968zeXR9 Mertens Keynote MN MESI March 2015 34 3/11/2015

Slide 35

Resources Mertens, D. M. & Wilson, A. (2012). Program Evaluation Theory and Practice: A Comprehensive Guide. NY: Guilford. Mertens, D. M. (2015). Research and evaluation in education and psychology: Integrating diversity with qual, quant and mixed methods. 4th ed. Thousand Oaks, CA: Sage. Mertens, D. M. (2009). Transformative research & evaluation. NY: Guilford. Mertens, D. M. & Ginsberg, P. (2009).(Eds.) Handbook of Social Research Ethics. Thousand Oaks, CA: Sage. American Evaluation Association (2011). Public Statement on Cultural Competence in Evaluation. AEA. Mertens Keynote MN MESI March 2015 35 3/11/2015

Slide 36

Contact information Donna M. Mertens, PhD Independent Consultant Donna.Mertens@Gallaudet.edu Mertens Keynote MN MESI March 2015 36 3/11/2015